Understanding the logic behind combinatorics and recursion solution (queries as comments in code)

Our local ninja Naruto is learning to make shadow-clones of himself and is facing a dilemma. He
only has a limited amount of energy (e) to spare that he must entirely distribute among all of his
clones. Moreover, each clone requires at least a certain amount of energy to function (m) . Your job is
to count the number of different ways he can create shadow clones.
Example:

e=7;m=2

ans = 4

The following possibilities occur:
Make 1 clone with 7 energy

Make 2 clones with 2, 5 energy

Make 2 clones with 3, 4 energy

Make 3 clones with 2, 2, 3 energy.

Note: <2, 5> is the same as <5, 2>.
Make sure the ways are not counted multiple times because of different ordering.

Answer

int count(int n, int k){
    if((n<k)||(k<1)) return 0;
    else if ((n==k)||(k==1)) return 1;
    else return count(n-1,k-1)+ count(n-k,k);   // logic behind this?
}

int main()
{
    int e,m;            // e is total energy and m is min energy per clone
    scanf("%d %d", &e, &m);
    int max_clones= e/m;
    int i,ans=0;
    for(i=1;i<=max_clones;i++){
        int available = e - ((m-1)*i);   // why is it (m-1)*i instead of m*i
        ans += count(available, i);
    }
    return 0;
}

query – Why does the underscore character interfere with Dataset queries?

Why does the underscore character affect the result of a Dataset query? Is there a workaround to this problem?

Here is a Dataset dsTable:

{
 {<|"gid" -> "1166182964626283", "name" -> "(no section)"|>},
 {<|"gid" -> "1166182964626284", "name" -> "SETUP"|>},
 {<|"gid" -> "1166182964626285", "name" -> "STRATEGY_VIEW "|>},
 {<|"gid" -> "1166182964626286", "name" -> "OPERATIONS_VIEW"|>},
 {<|"gid" -> "1166182964626287", "name" -> "TECH_VIEW"|>},
 {<|"gid" -> "1166182964626288", "name" -> "LEGAL_VIEW"|>},
 {<|"gid" -> "1166182964626289", "name" -> "FINANCE_VIEW"|>}
}

Now, when I query for “SETUP”…

dsTable(Select(#name == "SETUP" &), {"name", "gid"})

…I get a match, like so:

{<|"name" -> "SETUP", "gid" -> "1166182964626284"|>}

However, when I query for something with an UNDERSCORE in it..

dsTable(Select(#name == "STRATEGY_VIEW" &), {"name", "gid"}) 

I get no matches.

MS SQL Server 2012 – How To Find Last Executed Queries

I have 120+ databases and I want to find last executed queries (actually I want to find last executed queries and last executed select queries) in last 10 days. I have below scripts for this job:

1-) Some database names are NULL and there is no date.

SELECT
    txt.TEXT AS (SQL Statement),
    qs.execution_count (No. Times Executed),
    qs.last_execution_time AS (Last Time Executed), 
    DB_NAME(txt.dbid) AS (Database)
FROM SYS.DM_EXEC_QUERY_STATS AS qs
    CROSS APPLY SYS.DM_EXEC_SQL_TEXT(qs.SQL_HANDLE) AS txt
ORDER BY qs.last_execution_time DESC;

2-) There are no databases’ names and date.

SELECT dest.TEXT AS (Query),
deqs.execution_count (Count),
deqs.last_execution_time AS (Time)
FROM sys.dm_exec_query_stats AS deqs
CROSS APPLY sys.dm_exec_sql_text(deqs.sql_handle) AS dest
ORDER BY deqs.last_execution_time DESC;

3-)

SELECT deqs.last_execution_time AS (Time), dest.text AS (Query), dest.*
FROM sys.dm_exec_query_stats AS deqs
CROSS APPLY sys.dm_exec_sql_text(deqs.sql_handle) AS dest
WHERE dest.dbid = DB_ID('dbname')
ORDER BY deqs.last_execution_time DESC

4-) There is no database names.

SELECT        SQLTEXT.text, STATS.last_execution_time
FROM          sys.dm_exec_query_stats STATS
CROSS APPLY   sys.dm_exec_sql_text(STATS.sql_handle) AS SQLTEXT
WHERE         STATS.last_execution_time > GETDATE()-1
ORDER BY      STATS.last_execution_time DESC

5-) Below query runs but it’s too slow. It executing about 3 mins and it couldn’t even finish the part just for the last day.

SELECT 
sql_text.text, 
st.last_execution_time,
DB_NAME(qp.dbid) as databasename
FROM sys.dm_exec_query_stats st 
CROSS APPLY sys.dm_exec_sql_text(st.sql_handle) AS sql_text
INNER JOIN sys.dm_exec_cached_plans cp
ON cp.plan_handle = st.plan_handle
CROSS APPLY sys.dm_exec_query_plan(cp.plan_handle) as qp
WHERE st.last_execution_time >= DATEADD(week, -1, getdate())
ORDER BY last_execution_time DESC;

I want to order my columns like that: Text – Last Execution Time – DB Name

But I’m not an SQL man and I didn’t find any clear solution for this. How can I combine these scripts?

Regards,

Is there a search engine with reverse search mechanism: processing multiple queries and retrieving a single result for each of them?

A regular search engine retrieves a number of results matching a single query. I need one that does the opposite: enables to input a number of queries and displays a single match to each of them. If there is no match, I need to know it too. An implementation of this approach exists, but its application is limited to the database embedding it (PubMed). Batch Citation Matcher enables a user to identify unique content in databases covering material also present in PubMed. A user can submit a text file and the engine transforms the content of the file separated in a required way into a number of queries, then displays a table with queries in one column and the information on whether the text of the queries was found in a database being compared to PubMed in another column. Is there an analogues application which can be applied to any website?

azure sql database – How can I log which queries are in a distributed transaction on an MS SQL server?

I am looking at migrating a database from a self-hosted cluster to Microsoft Azure SQL. I am aware that there are a few distributed transactions involved, which isn’t supported on Azure SQL.

Is there a way that I can log all distributed transactions and their queries, so that I can inspect the client application and remove the requirement for distributed transactions?

mysql – Slow Database with concurrent queries, Table cache hitrate 0% [Newbie! Need help :)]

I’m a bit of a newbie when it comes to dba work so please take it easy!

I have gone through countless threads trying to improve the performance of my database, looking for someone who could give me some pointers on changes I may need to do for my variables that may squeeze more performance.
Or if I have misconfigured anything.

Here are the details
Server specs: E3 1270 V6 w/32gb Ram
OS: Windows Server 2016
Mysql Version: 8.0.19 – MySQL Community Server – GPL

MYSQL Config
https://ybin.me/p/b99f994ad62c27ad#zvwS1+XGP6ZIZtKYdiMOySg+aYR85Qp3ciTvFr6q4mE=

Show Global Status (during high usage):

https://ybin.me/p/b74fae252e807749#F9+88tWsVo/hqHjKlnlnG1gEUwj7vlonLlYrUxXnThg=

SHOW ENGINE INNODB STATUS (During High Usage)

InnoDB      
=====================================
2020-05-17 16:57:16 0xa6c INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 7 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 103908 srv_active, 0 srv_shutdown, 300397 srv_idle
srv_master_thread log flush and writes: 0
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 42855
OS WAIT ARRAY INFO: signal count 37923
RW-shared spins 3053, rounds 3116, OS waits 59
RW-excl spins 6909, rounds 59455, OS waits 1320
RW-sx spins 99, rounds 1306, OS waits 25
Spin rounds per wait: 1.02 RW-shared, 8.61 RW-excl, 13.19 RW-sx
------------------------
LATEST DETECTED DEADLOCK
------------------------
2020-05-17 13:52:00 0x2904
*** (1) TRANSACTION:
TRANSACTION 12175899, ACTIVE 0 sec starting index read
mysql tables in use 3, locked 3
LOCK WAIT 4 lock struct(s), heap size 1136, 3 row lock(s)
MySQL thread id 3340, OS thread handle 3660, query id 2398670 localhost 127.0.0.1 root updating
UPDATE user_inventory SET count = 22 WHERE identifier = 'steam:11000013f8eef2a' AND item = 'bandage'

*** (1) HOLDS THE LOCK(S):
RECORD LOCKS space id 584 page no 84 n bits 336 index PRIMARY of table `essentialmode`.`user_inventory` trx id 12175899 lock_mode X locks rec but not gap
Record lock, heap no 69 PHYSICAL RECORD: n_fields 6; compact format; info bits 0
 0: len 4; hex 8039a49f; asc  9  ;;
 1: len 6; hex 000000b9849a; asc       ;;
 2: len 7; hex 02000001ad0151; asc       Q;;
 3: len 21; hex 737465616d3a313130303030313366386565663261; asc steam:11000013f8eef2a;;
 4: len 7; hex 62616e64616765; asc bandage;;
 5: len 4; hex 8000002b; asc    +;;


*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 584 page no 2445 n bits 792 index item of table `essentialmode`.`user_inventory` trx id 12175899 lock_mode X waiting
Record lock, heap no 561 PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 0: len 7; hex 62616e64616765; asc bandage;;
 1: len 4; hex 800002e8; asc     ;;


*** (2) TRANSACTION:
TRANSACTION 12175886, ACTIVE 0 sec fetching rows
mysql tables in use 3, locked 3
LOCK WAIT 60 lock struct(s), heap size 8400, 2334 row lock(s)
MySQL thread id 3336, OS thread handle 10296, query id 2398648 localhost 127.0.0.1 root updating
UPDATE user_inventory SET count = 4 WHERE identifier = 'steam:11000010e1c050e' AND item = 'bandage'

*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 584 page no 2445 n bits 792 index item of table `essentialmode`.`user_inventory` trx id 12175886 lock_mode X
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
 0: len 8; hex 73757072656d756d; asc supremum;;

Record lock, heap no 561 PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 0: len 7; hex 62616e64616765; asc bandage;;
 1: len 4; hex 800002e8; asc     ;;


*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 584 page no 84 n bits 336 index PRIMARY of table `essentialmode`.`user_inventory` trx id 12175886 lock_mode X locks rec but not gap waiting
Record lock, heap no 69 PHYSICAL RECORD: n_fields 6; compact format; info bits 0
 0: len 4; hex 8039a49f; asc  9  ;;
 1: len 6; hex 000000b9849a; asc       ;;
 2: len 7; hex 02000001ad0151; asc       Q;;
 3: len 21; hex 737465616d3a313130303030313366386565663261; asc steam:11000013f8eef2a;;
 4: len 7; hex 62616e64616765; asc bandage;;
 5: len 4; hex 8000002b; asc    +;;

*** WE ROLL BACK TRANSACTION (1)
------------
TRANSACTIONS
------------
Trx id counter 12207888
Purge done for trx's n:o < 12207888 undo n:o < 0 state: running but idle
History list length 19
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 283609840283040, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 283609840282192, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 283609840281344, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: wait Windows aio (insert buffer thread)
I/O thread 1 state: wait Windows aio (log thread)
I/O thread 2 state: wait Windows aio (read thread)
I/O thread 3 state: wait Windows aio (read thread)
I/O thread 4 state: wait Windows aio (read thread)
I/O thread 5 state: wait Windows aio (read thread)
I/O thread 6 state: wait Windows aio (read thread)
I/O thread 7 state: wait Windows aio (read thread)
I/O thread 8 state: wait Windows aio (read thread)
I/O thread 9 state: wait Windows aio (read thread)
I/O thread 10 state: wait Windows aio (read thread)
I/O thread 11 state: wait Windows aio (read thread)
I/O thread 12 state: wait Windows aio (read thread)
I/O thread 13 state: wait Windows aio (read thread)
I/O thread 14 state: wait Windows aio (read thread)
I/O thread 15 state: wait Windows aio (read thread)
I/O thread 16 state: wait Windows aio (read thread)
I/O thread 17 state: wait Windows aio (read thread)
I/O thread 18 state: wait Windows aio (read thread)
I/O thread 19 state: wait Windows aio (read thread)
I/O thread 20 state: wait Windows aio (read thread)
I/O thread 21 state: wait Windows aio (read thread)
I/O thread 22 state: wait Windows aio (read thread)
I/O thread 23 state: wait Windows aio (read thread)
I/O thread 24 state: wait Windows aio (read thread)
I/O thread 25 state: wait Windows aio (read thread)
I/O thread 26 state: wait Windows aio (read thread)
I/O thread 27 state: wait Windows aio (read thread)
I/O thread 28 state: wait Windows aio (read thread)
I/O thread 29 state: wait Windows aio (read thread)
I/O thread 30 state: wait Windows aio (read thread)
I/O thread 31 state: wait Windows aio (read thread)
I/O thread 32 state: wait Windows aio (read thread)
I/O thread 33 state: wait Windows aio (read thread)
I/O thread 34 state: wait Windows aio (read thread)
I/O thread 35 state: wait Windows aio (read thread)
I/O thread 36 state: wait Windows aio (read thread)
I/O thread 37 state: wait Windows aio (read thread)
I/O thread 38 state: wait Windows aio (read thread)
I/O thread 39 state: wait Windows aio (read thread)
I/O thread 40 state: wait Windows aio (read thread)
I/O thread 41 state: wait Windows aio (read thread)
I/O thread 42 state: wait Windows aio (read thread)
I/O thread 43 state: wait Windows aio (read thread)
I/O thread 44 state: wait Windows aio (read thread)
I/O thread 45 state: wait Windows aio (read thread)
I/O thread 46 state: wait Windows aio (read thread)
I/O thread 47 state: wait Windows aio (read thread)
I/O thread 48 state: wait Windows aio (read thread)
I/O thread 49 state: wait Windows aio (read thread)
I/O thread 50 state: wait Windows aio (read thread)
I/O thread 51 state: wait Windows aio (read thread)
I/O thread 52 state: wait Windows aio (read thread)
I/O thread 53 state: wait Windows aio (read thread)
I/O thread 54 state: wait Windows aio (read thread)
I/O thread 55 state: wait Windows aio (read thread)
I/O thread 56 state: wait Windows aio (read thread)
I/O thread 57 state: wait Windows aio (read thread)
I/O thread 58 state: wait Windows aio (read thread)
I/O thread 59 state: wait Windows aio (read thread)
I/O thread 60 state: wait Windows aio (read thread)
I/O thread 61 state: wait Windows aio (read thread)
I/O thread 62 state: wait Windows aio (read thread)
I/O thread 63 state: wait Windows aio (read thread)
I/O thread 64 state: wait Windows aio (read thread)
I/O thread 65 state: wait Windows aio (read thread)
I/O thread 66 state: wait Windows aio (write thread)
I/O thread 67 state: wait Windows aio (write thread)
I/O thread 68 state: wait Windows aio (write thread)
I/O thread 69 state: wait Windows aio (write thread)
I/O thread 70 state: wait Windows aio (write thread)
I/O thread 71 state: wait Windows aio (write thread)
I/O thread 72 state: wait Windows aio (write thread)
I/O thread 73 state: wait Windows aio (write thread)
I/O thread 74 state: wait Windows aio (write thread)
I/O thread 75 state: wait Windows aio (write thread)
I/O thread 76 state: wait Windows aio (write thread)
I/O thread 77 state: wait Windows aio (write thread)
I/O thread 78 state: wait Windows aio (write thread)
I/O thread 79 state: wait Windows aio (write thread)
I/O thread 80 state: wait Windows aio (write thread)
I/O thread 81 state: wait Windows aio (write thread)
I/O thread 82 state: wait Windows aio (write thread)
I/O thread 83 state: wait Windows aio (write thread)
I/O thread 84 state: wait Windows aio (write thread)
I/O thread 85 state: wait Windows aio (write thread)
I/O thread 86 state: wait Windows aio (write thread)
I/O thread 87 state: wait Windows aio (write thread)
I/O thread 88 state: wait Windows aio (write thread)
I/O thread 89 state: wait Windows aio (write thread)
I/O thread 90 state: wait Windows aio (write thread)
I/O thread 91 state: wait Windows aio (write thread)
I/O thread 92 state: wait Windows aio (write thread)
I/O thread 93 state: wait Windows aio (write thread)
I/O thread 94 state: wait Windows aio (write thread)
I/O thread 95 state: wait Windows aio (write thread)
I/O thread 96 state: wait Windows aio (write thread)
I/O thread 97 state: wait Windows aio (write thread)
I/O thread 98 state: wait Windows aio (write thread)
I/O thread 99 state: wait Windows aio (write thread)
I/O thread 100 state: wait Windows aio (write thread)
I/O thread 101 state: wait Windows aio (write thread)
I/O thread 102 state: wait Windows aio (write thread)
I/O thread 103 state: wait Windows aio (write thread)
I/O thread 104 state: wait Windows aio (write thread)
I/O thread 105 state: wait Windows aio (write thread)
I/O thread 106 state: wait Windows aio (write thread)
I/O thread 107 state: wait Windows aio (write thread)
I/O thread 108 state: wait Windows aio (write thread)
I/O thread 109 state: wait Windows aio (write thread)
I/O thread 110 state: wait Windows aio (write thread)
I/O thread 111 state: wait Windows aio (write thread)
I/O thread 112 state: wait Windows aio (write thread)
I/O thread 113 state: wait Windows aio (write thread)
I/O thread 114 state: wait Windows aio (write thread)
I/O thread 115 state: wait Windows aio (write thread)
I/O thread 116 state: wait Windows aio (write thread)
I/O thread 117 state: wait Windows aio (write thread)
I/O thread 118 state: wait Windows aio (write thread)
I/O thread 119 state: wait Windows aio (write thread)
I/O thread 120 state: wait Windows aio (write thread)
I/O thread 121 state: wait Windows aio (write thread)
I/O thread 122 state: wait Windows aio (write thread)
I/O thread 123 state: wait Windows aio (write thread)
I/O thread 124 state: wait Windows aio (write thread)
I/O thread 125 state: wait Windows aio (write thread)
I/O thread 126 state: wait Windows aio (write thread)
I/O thread 127 state: wait Windows aio (write thread)
I/O thread 128 state: wait Windows aio (write thread)
I/O thread 129 state: wait Windows aio (write thread)
Pending normal aio reads: (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) , aio writes: (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) ,
 ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
25639 OS file reads, 5252794 OS file writes, 2142145 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 30, seg size 32, 78 merges
merged operations:
 insert 89, delete mark 3, delete 0
discarded operations:
 insert 0, delete mark 0, delete 0
Hash table size 4425293, node heap has 577 buffer(s)
Hash table size 4425293, node heap has 7 buffer(s)
Hash table size 4425293, node heap has 3 buffer(s)
Hash table size 4425293, node heap has 42 buffer(s)
Hash table size 4425293, node heap has 25 buffer(s)
Hash table size 4425293, node heap has 8 buffer(s)
Hash table size 4425293, node heap has 117 buffer(s)
Hash table size 4425293, node heap has 268 buffer(s)
63.71 hash searches/s, 4.29 non-hash searches/s
---
LOG
---
Log sequence number          8204611911
Log buffer assigned up to    8204611911
Log buffer completed up to   8204611911
Log written up to            8204611911
Log flushed up to            8204611911
Added dirty pages up to      8204611911
Pages flushed up to          8204611911
Last checkpoint at           8204611911
2113740 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 17582522368
Dictionary memory allocated 1618261
Buffer pool size   1048576
Free buffers       1010227
Database pages     37302
Old database pages 13779
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1961, not young 12509
0.00 youngs/s, 0.00 non-youngs/s
Pages read 25245, created 12063, written 2341874
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 37302, unzip_LRU len: 0
I/O sum(0):cur(0), unzip sum(0):cur(0)
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size   131072
Free buffers       126195
Database pages     4738
Old database pages 1766
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 47, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 3177, created 1561, written 390400
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4738, unzip_LRU len: 0
I/O sum(0):cur(0), unzip sum(0):cur(0)
---BUFFER POOL 1
Buffer pool size   131072
Free buffers       126202
Database pages     4747
Old database pages 1732
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 57, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 3144, created 1603, written 377393
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4747, unzip_LRU len: 0
I/O sum(0):cur(0), unzip sum(0):cur(0)
---BUFFER POOL 2
Buffer pool size   131072
Free buffers       126359
Database pages     4584
Old database pages 1682
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1596, not young 12509
0.00 youngs/s, 0.00 non-youngs/s
Pages read 3159, created 1431, written 286695
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4584, unzip_LRU len: 0
I/O sum(0):cur(0), unzip sum(0):cur(0)
---BUFFER POOL 3
Buffer pool size   131072
Free buffers       126376
Database pages     4563
Old database pages 1679
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 23, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 3060, created 1503, written 217282
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4563, unzip_LRU len: 0
I/O sum(0):cur(0), unzip sum(0):cur(0)
---BUFFER POOL 4
Buffer pool size   131072
Free buffers       126427
Database pages     4515
Old database pages 1683
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 39, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 3113, created 1402, written 296757
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4515, unzip_LRU len: 0
I/O sum(0):cur(0), unzip sum(0):cur(0)
---BUFFER POOL 5
Buffer pool size   131072
Free buffers       126346
Database pages     4604
Old database pages 1681
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 48, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 3120, created 1484, written 220370
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4604, unzip_LRU len: 0
I/O sum(0):cur(0), unzip sum(0):cur(0)
---BUFFER POOL 6
Buffer pool size   131072
Free buffers       126053
Database pages     4883
Old database pages 1815
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 104, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 3276, created 1607, written 259821
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4883, unzip_LRU len: 0
I/O sum(0):cur(0), unzip sum(0):cur(0)
---BUFFER POOL 7
Buffer pool size   131072
Free buffers       126269
Database pages     4668
Old database pages 1741
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 47, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 3196, created 1472, written 293156
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4668, unzip_LRU len: 0
I/O sum(0):cur(0), unzip sum(0):cur(0)
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Process ID=9652, Main thread ID=00000000000028B4 , state=sleeping
Number of rows inserted 425605, updated 1126794, deleted 24758, read 3734292042
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
Number of system rows inserted 1841, updated 1056, deleted 1656, read 1362568
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 69.28 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================

MYSQL Tuner Tips
https://i.gyazo.com/107f94f2e69fafe58d6f2125ff3c9ccc.png

I would highly appreciate any input on how we can make the database happier!
If I’m missing any information, let me know happy to provide any extra info!

Architecture – How can I increase the number of http queries made by PHP queue workers?

I am building an uptime monitoring service that consists of many nodes that send http requests to different hosts.

Use Laravel and its queues as a framework. First I send jobs to the queue, for example 400 jobs for 400 hosts. Then queue workers access these jobs and make http requests.

My goal is to be able to process a significant number of http requests in one minute. By significant I mean starting from 1000 and being able to expand this number to 5000-10000, for example.

I use digital ocean droplets for knots. The number of queries that can be processed depends on where the node is located.

Let's say we have a droplet in India. It has 2 shared CPU and 2 GB memory. If I run 25 to 30 queue employees who make requests (Laravel + Guzzle, a wrapper for cURL), it can hardly handle 400 to 500 hosts.

I can buy more CPU and more memory and process more in this way. However, I'm looking for ways to optimize the code and the process myself.

What I'm thinking about:

  1. Update to PHP 7.4 (7.2 is currently used, which is a bit slower in my opinion)
  2. Use native PHP instead of Laravel. However, it makes development a little more difficult because I use a lot of Laravel stuff
  3. Use cURL without consumption
  4. I don't have much knowledge about other technologies, but I think when I write http requests that are processed in another language (nodejs, go?), It can be faster

How I understand it:

  1. I cannot increase the request speed myself. If opening a host takes 3 seconds, I can't shorten the time. Note that I cannot use HEAD or OPTIONS requests because the user can choose the exact method. + I keep the response time, which is important

  2. I am limited to memory and CPU time. So I can somehow reduce memory and CPU usage while increasing the number of employees

  3. I can do something with multithreading. I am not currently using Guzzle multithreaded material because I have tested it before and it has been found that moving jobs in queues is quicker and easier

Any advice?

CSOM Caml queries only get root-level folders in the document library (paginated answer)

I can easily access the rootFolders of my test library on SharePoint Online

FolderCollection oFolderCollection = targetList.RootFolder.Folders;
clientContext.Load(oFolderCollection);
clientContext.ExecuteQuery();

The problem is that this doesn't work for my production library because it exceeds the view list threshold. I was able to work around the threshold error with the following code:

using (ClientContext clientContext = new ClientContext(webUri))
        {

            int rowLimit = 1;
            var camlQuery = new CamlQuery();
            camlQuery.ViewXml = @"" + rowLimit + "";

            clientContext.Credentials = new SharePointOnlineCredentials(userName, password);
            ListItemCollectionPosition position = null;

            // This value is NOT List internal name
            List targetList = clientContext.Web.Lists.GetByTitle("Clients");

            //1st attempt
            //FolderCollection oFolderCollection = targetList.RootFolder.Folders;
            //clientContext.Load(oFolderCollection);
            //clientContext.ExecuteQuery();

            int clientCount = 0;
            do
            {
                ListItemCollection listItems = null;
                camlQuery.ListItemCollectionPosition = position;
                listItems = targetList.GetItems(camlQuery);
                clientContext.Load(listItems);
                clientContext.ExecuteQuery();
                position = listItems.ListItemCollectionPosition;
                items.AddRange(listItems.ToList());

                foreach (ListItem item in listItems)
                {
                    if (item != null && item.Folder != null)
                    {
                        try
                        {
                            clientContext.Load(item);
                            clientContext.Load(item.Folder);
                            clientContext.Load(item.Folder.ParentFolder);
                            clientContext.ExecuteQuery();
                            if ((item != null && item.Folder != null) && item.Folder.GetType() == typeof(Folder))
                            {
                                clientCount++;



                                //currently getting all files and folders, just want to start at top level folder then load in others if/as needed
                                Console.WriteLine(clientCount + "- Client: " + item.Folder.Name);
                                //clientFileDiver(item.Folder, clientContext);

                            }
                        }
                        catch (Microsoft.SharePoint.Client.ServerException)
                        {
                            if(item == null || item.Folder == null)
                            {
                                continue;
                            }
                            throw;
                        }
                        catch (ServerObjectNullReferenceException)
                        {
                            if (item == null || item.Folder == null)
                            {
                                continue;
                            }
                        }
                        catch (Exception)
                        {
                            throw;
                        }

                    }

                }

            }
            while (position != null);

Try to ignore some of the ugly if statements: D. I'm very new to CSOM and Caml. Is it possible to write a caml statement that only returns the folders in the root of the document library? Or is there a way to change the original way I tried and restore the folders in a paginated response using the targetList.Rootfolder properties? Sorry if this is unclear! Happy Friday everyone!