performance tuning – Tricks to optimize a maximization problem

I am dealing with the following piece of code, to study a problem in quantum mechanics:

L(n_) := KirchhoffMatrix(CompleteGraph(n));

c(n_) := 1;
w(n_, p_) := Table(KroneckerDelta(k, p), {k, 1, n});
P(n_, p_) := KroneckerProduct(w(n, p), w(n, p));
s(n_) := 1/Sqrt(n)*Table(1, {k, 1, n});
Ps(n_) := KroneckerProduct(s(n), s(n));

H(n_, (Lambda)_) := (Lambda) L(n) - P(n, c(n)); 
U(n_, t_, (Lambda)_) := MatrixExp(-I*t*H(n, (Lambda)));
(Psi)(n_, t_, (Lambda)_) := U(n, t, (Lambda)).s(n);
prs(n_, t_, (Lambda)_) := Abs(w(n, c(n)).(Psi)(n, t, (Lambda)))^2;

Prob(n_) := NMaximize(prs(n, t, (Lambda)), {t,(Lambda)})((1))

The NMaximize function takes quite a while on my machine to compute $text{Prob}(n)$, so I would be interested in any suggestion to increase the efficiency of the code – taking into account that the input graph could be a different one. Probably the hardest part to compute is taking the exponential matrix of $H$, but I’m not aware of any way to optimize it.

performance – Python Avoid Loop with numpy

The problem is this:

I have a numpy array called “contour” of N coordinates (x,y) so of dimension (N , 2)

For each point in this table, I would like to create a square centered at that point and perform a test on the square formed.

I wanted to know if you had a way to solve my problem without using a for loop!

My version with a for loop:

def neighbour(point , mask  , n ): # Create the square around this point and count the number of neighbors.
      mask = mask(point(0) - int(n/2) : point(0) + int(n/2) + 1,point(1) - int(n/2):point(1) + int(n/2) + 1)
      return n**2 - np.count_nonzero(mask)



def max_neighbour(contour , mask=maske , n=ne): # Find the point with as many neighbors as possible
    t = np.zeros(len(contour)) # contour is the numpy array of dimension (2,N)
    for i in range(len(contour)):
        t(i) = neighbour(contour(i),mask,n)        
    return contour(np.argmax(t)) # t contains the list of neighbors for each contour point.

performance – vectorized crosstabulation in Python for 2 array with 2 category each

I have 2 python list label and presence. I want to do cross-tabulation AND get count for each block out of 4, such as A,B,C and D ine below code.

  • Both the lists have values True and False.
  • I have tried pandas crosstab function, however its slower than my code which is below.
  • One problem with my code is it’s not vectorized and is using a for loop which slows things.

Is there any way to make below function faster in python?

def cross_tab(label,presence):
    A_token=0
    B_token=0
    C_token=0
    D_token=0
    for i,j in zip(list(label),list(presence)):
        if i==True and j==True:
            A_token+=1
        elif i==False and j==False:
            D_token+=1
        elif i==True and j==False:
            C_token+=1
        elif i==False and j==True:
            B_token+=1
    return A_token,B_token,C_token,D_token

Some sample data and example input and output.

##input
label=(True,True,False,False,False,False,True,False,False,True,True,True,True,False)
presence=(True,False,False,True,False,False,True,True,False,True,False,True,False,False)

##processing
A,B,C,D=cross_tab(label,presence)
print('A:',A,'B:',B,'C:',C,'D:',D)

##Output
A: 4 B: 2 C: 3 D: 5

c# – How’d I enhance database read write performance

Friends, I have never ever set up scalable database. Moreover, I don’t have experience in writing code using C# and multiple DB. My end goal is to enhance DB performance and scale my website globally.

So, I understand theoretically I can do the following

  1. Have Master/slave DB combinations for many different reasons
  2. Setting up Redis
  3. Having load balancer
  4. Launch site to multiple servers and do Horizontal sharding etc.
  5. Read data from local slave DB/Redis
  6. Write data to central DB
  7. Read from DB directly, update key from Redis cache in case of any update/delete/insert operation etc.

But from all the above,

  1. I am not too clear what I said is right or wrong or how accurate it is
  2. I don’t know what should be the strategy from start to end
  3. If I use Redis, then how can I save data in it if am only familiar with RDBMS
  4. what plugins/Nuget to use
  5. what I need to do in case of disasters
  6. What are the code snippets to use
  7. what hosting service is right for me AWS/Bluehost etc.
  8. How can I use Amazon S3 to store images/videos etc. whats a code snippet I can refer
  9. How to perform aggregation in Redis such as count total users who commented in the post etc.
  10. what data structures to use and how to use with code snippet

Basically, above all, I want to set up something cheapest and most reliable, most efficient & safest system (well it’s just a state of mind but still handled by some external source like AWS)

I know alot of question but i think you got an idea what i am trying to achieve here

performance – Simple parser using flex and c++

This is an alternative parser based on the specifications from this question. Briefly stated, the input file is a text file which has at least 33 fields separated by semicolons.

If the fourth field begins with either a T or an E, the line is valid and a subset of it written to the output file. Specifically, fields as numbered from $0$, should be output in this order: $ {0, 2, 3, 4, 5, 6, 10, 9, 11, 7, 32}$, each separated by a comma. All other fields are discarded.

%{
#include <iostream>
#include <fstream>
#include <vector>
#include <string>
#include <algorithm>
#include <experimental/iterator>
#include <iterator>
#undef YY_DECL
#define YY_DECL int FileLexer::yylex()


class FileLexer : public yyFlexLexer {
public:
    FileLexer(std::istream& in, std::ostream& out) :
        yyFlexLexer{in, out},
        out{out}
    {}
    using FlexLexer::yylex;
    /// the yylex function is automatically created by Flex.
    virtual int yylex();
private:
    /// pointer to the current value
    std::vector<std::string> vec;
    std::ostream& out;
    unsigned fieldcount{0};
    bool valid{true};
};


%}
%option warn nodefault batch noyywrap c++
%option yyclass="FileLexer"

FIELD  (^;n)*
DELIM   ; 
%%

{DELIM} { }
n      {
            if (valid && fieldcount >= 33) {
                std::copy(vec.begin(), vec.end(), std::experimental::make_ostream_joiner(out, ","));
                out << 'n';
            }
            vec.clear();
            fieldcount = 0;
            valid = true;
            return 1;
        }
{FIELD} {
            if (valid) {
                switch (fieldcount++) {
                    case 0:
                    case 1:
                    case 4:
                    case 5:
                    case 6:
                    case 7:
                    case 9:
                    case 32:
                        vec.push_back(yytext);
                        break;
                    case 3:
                        if (yytext(0) == 'E' || yytext(0) == 'T') {
                            vec.push_back(yytext);
                            valid = true;
                        } else {
                            valid = false;
                        }
                        break;
                    case 10:
                        {
                            auto n{vec.size()};
                            vec.push_back(yytext);
                            std::iter_swap(vec.begin()+n, vec.begin()+n-2);
                        }
                        break;
                    case 11:
                        {
                            auto n{vec.size()};
                            vec.push_back(yytext);
                            std::iter_swap(vec.begin()+n, vec.begin()+n-1);
                        }
                        break;

                }
            }
        }
%%
int main(int argc, char *argv()) {
    if (argc >= 3) {
        std::ifstream in{argv(1)};
        std::ofstream out{argv(2)};
        FileLexer lexer{in, out};
        while (lexer.yylex() != 0)
        {}
    }
}

Compile with:

flex -o parsefile.cpp lexer.l 
g++ -O2 -std=gnu++17 parsefile.cpp -o parsefile

This works but is slow (2.165 s) on my machine, with the same million-line input file as mentioned in my answer to the other question.

I tried it a few different ways but was unable to get a version that was faster than the PHP code in the other question. The switch statement logic is arguably a bit overly clever and stores only the needed fields in the desired order, but the speed was about the same as the straightforward implementation.

If it matters, I’m using gcc version 10.1 and flex 2.6.4 on a 64-bit Linux machine.

postgresql – Postgres, trigger function, is it bad for performance?

I have added the following trigger function to one of my tables in my Postgres database.

CREATE OR REPLACE FUNCTION trigger_set_timestamp()
RETURNS TRIGGER AS $$
BEGIN
  NEW.updated_at = NOW();
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

I am new to this, so, I am wondering, is this going to slow down my database? Should I avoid trigger functions for performance reasons and if so, what’s the alternative?

(In this case this is the only way I knew how to make the updated_at date column always set the date when any of the columns in the table would change….)

performance – How to benchmark Switchable Optimizations in MySQL

I want to benchmark the performance of Switchable Optimizations in MySQL, i.e., comparing the query latency and the result of EXPLAIN with/without using these optimization switches.

I have already tried to use sysbench and MySQL official test-suite. But I fond that they are too general and mainly for functional testing, i.e., runing these tests with/without the optimization switches, the performance are almost identical. What I want is sth like a performance test suite.

Could you please suggest me some good tools/suites to fulfill my demands?
Thanks!

Postgresql update performance – Database Administrators Stack Exchange

When I try to update a table that has millions of tuples it takes forever.
The table has no indexes and I’m updating about 10k tuples on a boolean column using only the table id. This operation is done a lot of times.

update my_table set is_enabled=? where id in (? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? ,...

Is there a way to improve updates on Postgresql?

version: 11.6
table:

create table my_table
(
    id          serial not null
        constraint contingency_v2_pkey
            primary key,
    field1      varchar(15),
    field2 varchar(255),
    field3  numeric,
    field4  numeric,
    field5        numeric,
    field6        integer,
    updated_at  varchar(30),
    is_enabled  boolean
);

optimization – How to create index to improve performance of an aggregate function that creates a table in oracle

I am creating an Oracle ORDS API using APEX_JSON. I recently started using bind variables instead of string concatenation using ||. I am trying to use an in clause in my where condition.

The problems begin here. The field I need to have on the left side of in is a number and the parameter to my stored procedure needs to be varchar2 as it is a comma seperated list of numbers.

Example (edited for brevity)

CREATE OR REPLACE PROCEDURE GET_CATEGORYPRODS (
    PCATEGORYID IN NUMBER,
    COMMASEPPRODUCTIDS IN VARCHAR2
) AS

l_cursor               SYS_REFCURSOR;
v_stmt_str             STRING(5000);
v_name                 NUMBER; --PRODUCT.NAME%TYPE;
v_displayorder         NUMBER; --PRODUCTCATEGORY%TYPE;
BEGIN
 v_stmt_str := 'SELECT 
    P.NAME, 
    PC.DISPLAYORDER
FROM 
    PRODUCT P
INNER JOIN
    PRODUCTCATEGORY PC
ON P.PRODUCTID = PC.PRODUCTID
WHERE 
   PC.CATEGORYID := :CATEGORYID
AND
   (P.PRODUCTID IN (SELECT * FROM TABLE(STRING_TO_TABLE_NUM(:COMMASEPPRODUCTIDS))) -- PREVIOUSLY WHERE || OCCURRED
        OR (:COMMASEPPRODUCTIDS IS NULL))';

s_counter := 0;

OPEN l_cursor FOR v_stmt_str
        USING pcategoryid, commasepproductids, commasepproductids;

FETCH l_cursor INTO
    v_productid,
    v_displayorder;

APEX_JSON.OPEN_ARRAY;
LOOP
    EXIT WHEN l_cursor%notfound;
    apex_json.open_object;
    apex_json.write('ProductID', v_productid);
    apex_json.write('DisplayOrder', v_displayorder);
    apex_json.close_object;
END LOOP;
apex_json.close_all;

END GET_CATEGORYPRODS;

Sample of parameters
'97187,142555,142568,48418,43957,44060,45160,45171,333889,333898'

To handle this problem, I created an aggregate function that takes in a string, splits on the commas, and pipes the row to a custom type.

Custom Type

create or replace type tab_number is table of number;

Aggregate Function

create or replace FUNCTION string_to_table_num (
    p VARCHAR2
)

   RETURN tab_number
   PIPELINED IS
BEGIN
   FOR cc IN (SELECT rtrim(regexp_substr(str, '(^,)*,', 1, level), ',') res
                FROM (SELECT p || ',' str FROM dual)
              CONNECT BY level <= length(str) 
                                  - length(replace(str, ',', ''))) LOOP
      PIPE ROW(lower(cc.res));
   END LOOP;
    
END;

The query slowed down significantly. I figured some optimization was needed but I had never done any sort of optimization before. After some research, I found EXPLAIN PLAN and ran it on the orginal query. I couldn’t get a good answer because of the bind variables, so I decided to run it on the aggregate function.

EXPLAIN PLAN QUERIES

explain plan for select * from TABLE(string_to_table_num('97187,142555,142568,48418,43957,44060,45160,45171,333889,333898'));

SELECT * 
FROM   TABLE(DBMS_XPLAN.DISPLAY);

When I ran EXPLAIN PLAN for the aggregate function the results were:

Plan hash value: 127161297
 
---------------------------------------------------------------------------------------------------------
| Id  | Operation                         | Name                | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                  |                     |  8168 | 16336 |    29   (0)| 00:00:01 |
|   1 |  COLLECTION ITERATOR PICKLER FETCH| STRING_TO_TABLE_NUM |  8168 | 16336 |    29   (0)| 00:00:01 |
---------------------------------------------------------------------------------------------------------

As I stated before, I am a noob to analyzing and optimizing queries, but 8168 Rows and 16336 bytes seems to be a lot for such a simple function. I looked into it, and found that the problem may be the lack of indexing of the pipelined table. I tried to add an index to the type tab_number but it turned it into a PL/SQL object that needed to be declared in a query, not a function.

I am pretty lost with this one. If you have any suggestions for any of the scenarios I mentioned, I am all ears. Thanks in advance.

performance – Is there any program to check how many my program eat resources?

Is there a program to check how my program is resource-hungry? It’s very simple, two loops with other functions inside. On my and my friends computers it works good, but I got info from other users they got frame-dropps while using my program. So, there is program to check how many resource my program eat on that low scale?