Parallelism – time complexity of Parallel.ForEach

Pseudocode and comments to explain:

// first select companies to process from db

foreach (company) {
    // select the employees
    foreach (employee of company) {
        // select items they can access
        foreach (item) {
           // do some calculation and save

I think this will be an O (n ^ 3) time complexity. Please correct me if I am wrong. Big O always gave me a headache. My question is, if you introduce parallel processing at the first level, what will it be? What if we introduce a second? parallel.foreach() Also for the second iteration?

SQL Server – Parallelism in a specific database is not active

I have a few databases with identical schemas, let's call them:

DB_A

DB_B

I have a query that takes 11 seconds to execute and returns 11,000 records in DB_A. However, if exactly the same query is executed in the same DB_B, it takes 40 seconds and returns 7,000 records.

The schema is identical, so the query is, but when I run it on DB_B, it goes to degree of parallelism 1, on DB_A it goes to 16.

I tried to set the threshold cost to 0 to enforce concurrency, but I got the same result.

Why is that? How can the same query that is executed in cloned databases behave so differently?

Any ideas are welcome.

I use SQL 2017 standard.

Parallelism – How is this simulation parallelized in Python?

I have a Dungeon and Dragons 5th Edition simulation game. Players, enemies, etc. are represented with Python code. I have a simulation class and the key part is here.

def run(self, n=10000):
"""
Run the Encounter n times, then print the results
"""
self.reset_stats()
for _ in range(n):
    self._encounter.run()
    self.collect_stats()
    self._encounter.reset()
self.calculate_aggregate_stats(n)
self.print_aggregate_stats()

Each simulation has its own Encounter Example. When you run the simulation, do it encounter multiple times. After every run of the encounter, you collect stats (A dictionary that you put in an instance variable of Simulation) and then put the encounter because that encounter and the objects it contains are changed in the course of a run. After you've done all the runs encounter, calculate the overall statistics (basically the average of all statistics in self._statswhich is why you have to surrender n).

I want to parallelize this code so that the n Times through the loop are distributed among multiple threads / processes. If necessary to avoid competition conditions, I can just make a copy of it encounter and change self._stats from an instance variable to a local variable (and I would change that calculate_aggregate_stats appropriate).

How do I do that in parallel? I have experience with MPI, OpenMP, and Pthreads in C and C ++, but I do not understand how parallel processing works in Python. I suspect I should do multiprocessing instead of multithreading because of GIL, but how do I do multiprocessing?

(If this question is better for another platform (eg StackOverflow), please let me know, I just assumed that it is a design question that should be answered here.)

What should we learn / use about Java parallelism in 2019?

I have been learning Java for more than a year. However, I only hear about concurrency and have never seen it in practice. Therefore, I have no idea which method is best / most commonly used. Examples:

  • Thread classes
  • Parallel stream
  • Future / completeable future

I thought I would be "cool" if I used Parallel Stream in an API call, but after seeing this: https://dzone.com/articles/think-twice-using-java-8 I'm mine not sure anymore what to do. Learn to use in Java for parallelism

How is parallelism handled during an INSERT SELECT cycle?

The application should restrict access to a specific resource to a maximum number of accesses every 24 hours. To track this, we have the following table:

CREATE TABLE accesses (
    user_id         INTEGER NOT NULL,
    dt              TIMESTAMPZ
);

Each time you access the resource, we record it in the table. However, we also need to check if the access limit has been reached. For this purpose, we need to execute this query:

SELECT count(*) 
FROM accesses 
WHERE dt >= timezone('utc', now()) - interval '24 hours' AND user_id = $1

and we have to compare the result with the user limit.

If two requests are requested at the same time, problems may occur:

  • When we first read, compare, and insert, we can grant more hits than the limit.
  • If we insert first, then read and compare, we can not allow legitimate access.

The two possibilities I know are either a SERIALIZABLE Transaction or SELECT ... FOR UPDATE, I am not sure if the latter actually works in this case, because we do not update the read lines, but insert new ones. So I think it will be a useless lock and the result will probably still be wrong.

With a SERIALIZABLE Transaction I tried again if it fails.

What should be done in this case to ensure proper access restrictions?

Java Best Way (DS / Algorithm) to develop the problem of maximum parallelism API counting

You get a list of data with the following attributes: start time, rest API name / service name, end time. You have to find the maximum parallelism that has been reached. Example – {{1, A, 4}, {2, B, 3}, {4, C, 10}, {4, D, 7}, {2, E, 4}}.

The answer here is 4, because at time t = 4 4 services are executed, namely A, C, D, E.

=> Here I can create an array of 24 (hour) * 60 (minute) * 60 (second). When I get an API, I update the entire index every second. I have to do that for every API. Lastly, I can iterate over array and print maximum time if it had the highest Api count.

but I am not satisfied with this approach. What can I do to make it better?

Problems with the parallelism

Thank you for your attention.

I came here with a problem and I really have no idea how to overcome it.

The problem is that a consumer who stores his articles at a fixed interval sometimes has irrelevant articles.

It's time for details.
I deliberately work on my very first .NET application to learn how to work with tasks. Is it about the calculation of the faculty.

And the user interface looks like this:

Enter image description here

When the user enters the number, the calculation process begins:

Enter image description here

The CalculatorViewModel monitors the results at a specified interval. The results will be in the CalculaionsProvider this represents inherently data virtualization patterns.

These CalculationsProvider is subscribed to several events by FactorialCalculationService, The service has a concurrent queue where intermediate results are stored. These results are driven by FactorialCalculatorService from different threads (depends on how many cores your machine has).

Here is the scheme of interoperability between the components (for the sake of brevity I have excluded CalculatorViewModel since it does not matter here):

Enter image description here

I hope it makes sense how it works. Now that you understand what an application looks like, it's time to spot the problem.

Imagine, I have decided to calculate the faculty of the number 10000, and the calculation process expires:

Enter image description here

But suddenly I realized that this is not what I really want to do, because I want to calculate the faculty of the number 20. I enter this value and press the button.

And I expect that I will see that the result will be the number 20 calculation process. And sometimes it is:

Enter image description here

But sometimes it is not:

Enter image description here

In the picture above, you can see that there are some besides the number 20 calculation process Overhead cost calculations from the confidential calculation process of the number 10000, And I know where the problem is, but I do not know how to fix it.

I'll tell you where the problem is and why it is and then give you the code CalculationsProvider, FactorialCalculationService, FactorialCalculatorService.

This is because the consumer (CalculationsProvider) received his share of the articles just prior to the revocation and is on the way to move it to his local store (list only). However, another calculation was started. And the consumer's ClearCalculations () method was called before that part of the elements was moved. Storing the consumer locally rather than being empty before a new calculation already contains some irrelevant elements from the previous one. And if the consumer was not up to date before the revocation, everything works fine.

So that's a problem. If you are still here and have worked through all this text, you are incredible! Many thanks.

Here comes the code.

Calculations provider:

// 
/// Consumes and stores intermediate results of the faculty calculation.
/// Used when virtualizing the collection.
/// 
public class CalculationsProvider: IPagedSourceProviderAsync
{
private IModelFactory _modelFactory;
private IList _berechnungen;
private ReaderWriterLockSlim _readerWriterLockSlim;
private IFactorialCalculationService _factorialCalculationService;

public CalculationsProvider (IModelFactory modelFactory,
IFactorialCalculationService factorialCalculationService)
{
_modelFactory = modelFactory;
_calculations = new SortedList();
_readerWriterLockSlim = new ReaderWriterLockSlim ();
_factorialCalculationService = factorialCalculationService;

_factorialCalculationService.OnCalculationStarted + = () => ClearCalculations ();
_factorialCalculationService.OnDataReceived + = () => AddInterimCalculations ();
}

#region Implementation IPagedSourceProvider

    public bool IsSynchronized => throw new NotImplementedException ();

public object SyncRoot => throw new NotImplementedException ();

int IPagedSourceProvider.IndexOf (ICalculationModel element)
{
return _calculations.IndexOf (item);
}

int IPagedSourceProvider.Number
{
get {return _calculations.Count; }
}

PagedSourceItemsPacket IPagedSourceProvider.GetItemsAt (int pageoffset,
count int,
bool usePlaceholder)
{
Return new PagedSourceItemsPacket()
{
LoadedAt = DateTime.Now,
Items = _calculations.Skip (pageoffset) .Take (count)
};
}
#endregion

#region Implementation IPagedSourceProviderAsync

    public task GetCountAsync ()
{
return Task.Run (() =>
{
return this._calculations.Count;
});
}

public task <PagedSourceItemsPacket> GetItemsAtAsync (int pageoffset, int count, bool usePlaceholder)
{
return Task.Run (() =>
{
_readerWriterLockSlim.EnterWriteLock ();

PagedSourceItemsPacket page =
new PagedSourceItemsPacket()
{
LoadedAt = DateTime.Now,
Items = _calculations.Skip (pageoffset) .Take (count) .ToList ()
};

_readerWriterLockSlim.ExitWriteLock ();

Return page;
});
}

public ICalculationModel GetPlaceHolder (int index, int page, int offset)
{
var placeholderModel = _modelFactory.CreateCalculationModel ();

placeholderModel.InputNumber = "";
placeholderModel.Factorial = "";

return placeholderModel;
}


public task IndexOfAsync (ICalculationModel element)
{
return Task.Run (() => this._calculations.IndexOf (item));
}

public void OnReset (int count)
{
//
}
#endregion

/// 
    /// Adds intermediate results to the list
/// 
    Protected Void AddInterimCalculations ()
{

List <tuple> Intermediate calculations;

if (_factorialCalculationService.TryGetInterimCalculations (out interimCalculations, 100))
{
List <Task> tasks = new list <task> ();

foreach (var intCalc in interimCalculations)
{
tasks.Add (Task.Run (() =>
{
ICalculationModel interimCalculation = _modelFactory.CreateCalculationModel ();

interimCalculation.InputNumber = intCalc.Item1.ToString ();
interimCalculation.Factorial = intCalc.Item2.ToString ();

return interimcalculation;
}));
}

Task.WaitAll (tasks.ToArray ());

_readerWriterLockSlim.EnterWriteLock ();

foreach (var task in tasks)
{
_calculations.Add (task.Result);
}

_readerWriterLockSlim.ExitWriteLock ();
}
}

private void InitializeProvider ()
{

}

private void ClearCalculations ()
{
_readerWriterLockSlim.EnterWriteLock ();
_calculations.Clear ();
_readerWriterLockSlim.ExitWriteLock ();
}
}

FactorialCalculatonService:

public class FactorialCalculationService: IFactorialCalculationService
{
private system timer _timer;
private volatile bool _istCanceled;
private volatile bool _isInPorcess;
private CancellationTokenSource _tokenSourse;
private ConcurrentQueue <tuple> _interimCalculationQueue;
private read-only IFactorialCalculatorService _factorialCalculatorService;

/// Performs calculations and provides intermediate results
    public FactorialCalculationService (IFactorialCalculatorService factorialCalculatorService)
{
_isCanceled = false;
_isInPorcess = false;
_factorialCalculatorService = factorialCalculatorService;
_interimCalculationQueue = new ConcurrentQueue <Tuple> ();
}


private event action _onCalculationStopped;
private event action _onCalculationStarted;
private event action _onDataReceived;

public event Action OnCalculationStopped
{
Add
{
_onCalculationStopped + = value;
}
Clear
{
_onCalculationStopped - = value;
}
}

public event Action OnCalculationStarted
{
Add
{
_onCalculationStarted + = value;
}
Clear
{
_onCalculationStarted - = value;
}
}

public event action OnDataReceived
{
Add
{
_onDataReceived + = value;
}
Clear
{
_onDataReceived - = value;
}
}

#region IFactorialCalculationService implementation

public bool IsInPorcess => _isInPorcess;

public bool TryGetInterimCalculations (out List <Tuple> Intermediate results,
int countRecords)
{
if (_isCanceled ||! _isInPorcess)
{
interimResults = null;
return it incorrectly;
}

if (countRecords> _interimCalculationQueue.Count)
{
countRecords = _interimCalculationQueue.Count;
}

int i = 0;
tuple Items;
interimResults = new list <tuple> ();

while I < countRecords && _interimCalculationQueue.TryDequeue(out item))
        {
            interimResults.Add(item);
            i++;
        }

        return interimResults.Count != 0;
    }

    public Task StartCalculation(BigInteger number)
    {
        return 
            Task.Run(
                () =>
                {
To attempt
{
while (_isInPorcess);

StartNotification ();
_onCalculationStarted ();

_isInPorcess = true;
_tokenSourse = new CancellationTokenSource ();

_factorialCalculatorService.Calculate (
Number,
(n, res) => _interimCalculationQueue.Enqueue (
new tuple(n, res)
)
_tokenSourse.Token
);


// Wait until all results have been used up
while (! _interimCalculationQueue.IsEmpty);
}
catch (AggregateException ax)
{
ax.Handle (ex => {
return ex is OperationCanceledException;
});
}
Finally
{
_interimCalculationQueue = new ConcurrentQueue <Tuple> ();
_isInPorcess = false;
}
}
);
}

Public Void Stop Calculation ()
{
if (! _isCancelled)
{
Stop the notification ();
_tokenSourse.Cancel ();
}
}

#endregion

private void StartNotification ()
{
_timer = new SystemTimer () {Interval = 500};
_timer.Elapsed + = (o, e) => _onDataReceived ();
_timer.Start ();
}

private void StopNotification ()
{
_timer.Stop ();
}

}

FactorialCalculatorService

/// 
/// Calculates the faculty
/// in separate threads
/// 
public class FactorialCalculatorService: IFactorialCalculatorService
{
private int _maxTasks;

#region IFactorialCalculatorService implementation

/// 
    /// Calculates the faculty of the number
/// Uses a series of tasks that perform calculations to calculate the final result.
/// 
    /// The number of goals whose faculty must be calculated
    /// 
    /// callback method gives you intermediate results.
/// The first parameter of this represents the target intermediate number.
/// The second represents the faculty.
/// 
    /// Pass your own token to stop the calculation
    /// 
    
    
    
    public BigInteger Calculate (BigInteger number,
action Callback = null,
CancellationToken CancellationToken = Default (CancellationToken)
{
if (number < 0)
            throw new ArgumentOutOfRangeException(nameof(number));

        _maxTasks = number > Environment.ProcessorCount? Environment.ProcessorCount: 1;

var tasks = ParallelizeCalculation (Number, Callback, CancellationToken);

Task.WaitAll (Tasks);

return GetFinalResult (tasks);
}

#endregion

/// 
    /// Multiply all members within this number class.
/// 
    /// 
    
    
    
    /// 
    
    
    
    /// Cancels the calculation
    /// Returns intermediate results within this number class
    /// 
    
    
    
    Protected virtual BigInteger CalculateClass (BigInteger startFrom,
BigInteger upperBound,
CancellationToken CancellationToken = Default (CancellationToken),
action Callback = null)
{
CancellationToken.ThrowIfCancellationRequested ();

BigInteger internalResult = BigInteger.Zero;

for (var i = startFrom; i <= upperBound; i + = _maxTasks)
{
CancellationToken.ThrowIfCancellationRequested ();

internalResult = compute (i);
Recall? .Invoke (i, internalResult);
}

return internalResult;
}

/// 
    /// Calculates the faculty of the number
/// 
    /// 
    
    
    
    /// 
    
    
    
    protected BigInteger virtual calculation (BigInteger number)
{
Big integer result = 1;

for (BigInteger i = result; i <= number; i ++)
{
Result * = i;
}

Return result;
}

/// 
    /// Divides the calculation of the number into several tasks
/// 
    /// 
    
    
    
    /// 
    
    
    
    /// 
    
    
    
    /// Tasks that perform a calculation
    protected virtual task[]    ParallelizeCalculation (BigInteger number,
action Callback = null,
CancellationToken CancellationToken = Default (CancellationToken)
{
var tasks = Enumerable.Range (0, _maxTasks)
.Choose(
i => Task.Factory.StartNew (
() => CalculateClass (i, number, CancellationToken, Callback),
TaskCreationOptions.AttachedToParent
)
)
.ToArray ();


Return tasks;
}

/// 
    /// Multiplies the intermediate results in the specified order
/// to return the final result
/// 
    /// Tasks that perform a calculation
    /// 
    
    
    
    private BigInteger GetFinalResult (Task[]    Tasks)
{
BigInteger finalResult = 1;

foreach (var task in tasks)
{
finalResult * = task.Result;
}

return finalResult;
}
}

I may have implemented it incorrectly or it is completely wrong. I will be grateful for the advice. I have to use a different pattern if I want to implement it.

Many thanks!

SharePoint List Item Update – Parallelism using the REST API

I use the REST API to update the SharePoint list item (counter). In my code loading the form, I will retrieve the number from the list and increase it to 1 and click the button when sending to update the value of the number. I have reached this far.

Problem occurs when two users send the form at the same time, instead the counter value in the list is incremented by 1 to 2.

Checked with tags. But I noticed that during the post, if etag does not match get request etag, an error will be issued.

Is there a way to achieve this functionality using the REST API and properly increment the counter if more than 3-4 users submit the form?