python 3.x – Multiple HTTP requests with threading and queues

Im working on a I/O bounds application where I want to learn how to use threading property as well as queues to minimize the CPU usage as well as RAM resources. My plan was to use threading queues to do it and this is what I have done so far

# System modules
import time
from queue import Queue
from threading import Thread
from loguru import logger
import requests

feed_urls = (
    'http://www.foxnews.com/',
    'http://www.cnn.com/',
    'http://europe.wsj.com/',
)

# Set up some global variables
num_fetch_threads = 5
queue_exploring = Queue()
queue_monitoring = Queue()
save_length = {}


def get_requests(url):
    return len(requests.get(url).text)


def send_notifcation(queue: Queue):
    while True:
        url = queue.get()
        logger.info(f'Sending notifications: {url}')
        # FIXME Send notifications
        queue.task_done()


def explore_links(queue: Queue):
    """This is the worker thread function.
    It processes items in the queue one after
    another.  These daemon threads go into an
    infinite loop, and only exit when
    the main thread ends.
    """
    while True:
        url = queue.get()
        get_data_length = get_requests(url)

        if save_length(url) != get_data_length:
            logger.info(f"New changes on the webpage found! -> {url}")
            queue_monitoring.put(url)

        logger.info(f"No new changes in the page found! -> {url}")
        time.sleep(60)

        # Add back to queue
        queue.put(url)
        queue.task_done()


for i in range(num_fetch_threads):
    worker_one = Thread(target=explore_links, args=(queue_exploring,))
    worker_two = Thread(target=send_notifcation, args=(queue_monitoring,))

    worker_one.setDaemon(True)
    worker_two.setDaemon(True)
    worker_one.start()
    worker_two.start()


def main():
    logger.info('*** Main thread waiting ***')
    for url in feed_urls:
        response = get_requests(url)
        save_length(url) = response
        queue_exploring.put(url)

    queue_exploring.join()
    queue_monitoring.join()
    logger.info('*** Done ***')


if __name__ == '__main__':
    main()

The idea is that we want forever loop and see if a webpage has done any changes and if they have done any changes, we want to be notified that there has been a change. Simple as that. However here I do use multiple threads as well as queues, one for monitoring to see if there has been a change in the requests and the second one is to send notification

java – Concurrency: Server that deduplicates 9-digit numbers from multiple clients and writes them to a log

Problem Statement:

Write a server (“Application”) in Java that opens a socket and restricts input to at most 5 concurrent clients. Clients will connect to the Application and write any number of 9 digit numbers, and then close the connection. The Application must write a de-duplicated list of these numbers to a log file in no particular order.

Primary Considerations

        • The Application should work correctly as defined below in Requirements.
        • The overall structure of the Application should be simple.
        • The code of the Application should be descriptive and easy to read, and the build method and runtime parameters must be well-described and work.
        • The design should be resilient with regard to data loss.
        • The Application should be optimized for maximum throughput, weighed along with the other Primary Considerations and Requirements below.

Requirements

        1. The Application must accept input from at most 5 concurrent clients on TCP/IP port 4000.
        2. Input lines presented to the Application via its socket must either be composed of exactly nine decimal digits (e.g.: 314159265 or 007007009) immediately followed by a server-native newline sequence; or a termination sequence as detailed in #9, below.
        3. Numbers presented to the Application must include leading zeros as necessary to ensure they are each 9 decimal digits.
        4. The log file, to be named "numbers.log”, must be created anew and/or cleared when the Application starts.
        5. Only numbers may be written to the log file. Each number must be followed by a server-native newline sequence.
        6. No duplicate numbers may be written to the log file.
        7. Any data that does not conform to a valid line of input should be discarded and the client connection terminated immediately and without comment.
        8. Every 10 seconds, the Application must print a report to standard output:
                • The difference since the last report of the count of new unique numbers that have been received.
                • The difference since the last report of the count of new duplicate numbers that have been received.
                • The total number of unique numbers received for this run of the Application.
                • Example text for #8: Received 50 unique numbers, 2 duplicates. Unique total: 567231
        9.If any connected client writes a single line with only the word "terminate" followed by a server-native newline sequence, the Application must disconnect all clients and perform a clean shutdown as quickly as possible.
        10. Clearly state all of the assumptions you made in completing the Application along with any instructions on how to set up and run it in a README file.

Notes

        • You may write tests at your own discretion. Tests are useful to ensure your Application passes Primary Consideration A.
        • You may use common libraries in your project such as Apache Commons and Google Guava, particularly if their use helps improve Application simplicity and readability. However the use of large frameworks, such as Akka, is prohibited.
        • Your Application may not for any part of its operation use or require the use of external systems, for example Apache Kafka or Redis.
        • At your discretion, leading zeroes present in the input may be stripped—or not used—when writing output to the log or console.
        • Robust implementations of the Application typically handle more than 2M numbers per 10-second reporting period on a modern MacBook Pro laptop (e.g.: 16 GiB of RAM and a 2.5 GHz Intel i7 processor).

My implementation:

Application.java:

public class Application {
    static final int PORT = 4000;

    public static void main(String() args) {
        Server server = new Server();
        System.out.println("Starting server");
        server.start(PORT);
    }
}

Server.java:

import java.io.*;
import java.net.ServerSocket;
import java.net.Socket;
import java.net.SocketException;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.regex.Pattern;

public class Server {
    private static final int MAX_CLIENTS = 5;
    private ServerSocket serverSocket;
    private ExecutorService executorService = Executors.newFixedThreadPool(MAX_CLIENTS);

    private FileWriter fw;
    private BufferedWriter bw;
    BlockingQueue blockingQueue = new LinkedBlockingQueue();


    public void start(int port) {
        try {
            serverSocket = new ServerSocket(port);
        } catch (IOException e) {
            e.printStackTrace();
        }

        try {
            fw = new FileWriter("numbers.log", false);
        } catch (IOException e) {
            e.printStackTrace();
        }
        bw = new BufferedWriter(fw);
        LogWriterTask logWriterTask = new LogWriterTask(blockingQueue, bw);
        logWriterTask.start();


        while (true) {
            try {
                ClientHandler clientHandler = new ClientHandler(serverSocket.accept(), blockingQueue);
                executorService.submit(clientHandler);
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }

    private class ClientHandler extends Thread {
        private Socket clientSocket;
        private BufferedReader in;
        BlockingQueue<Integer> blockingQueue;
        private Pattern numberPattern = Pattern.compile("\d{9}");

        public ClientHandler(Socket socket, BlockingQueue<Integer> blockingQueue) {
            this.clientSocket = socket;
            this.blockingQueue = blockingQueue;
        }

        @Override
        public void run() {

            try {
                in = new BufferedReader(
                        new InputStreamReader(clientSocket.getInputStream()));

                String inputLine = "";

                System.out.println("Client connection started");
                while (true) {

                    try {
                        inputLine = in.readLine();
                        if (inputLine == null) {
                            break;
                        }
                        if (inputLine.equals("terminate")) {    //Disconnect all clients
                            System.exit(0);
                        };
                    } catch (SocketException e) {
                        e.printStackTrace();
                        stopClient();
                        break;
                    }

                    int num = processNumber(inputLine);
                    if (num == -1) {
                        stopClient();
                        break;
                    }
                    blockingQueue.add(num);
                }
            } catch (IOException e) {
                e.printStackTrace();
            } finally {
                stopClient();
            }
        }


        public void stopClient() {
            try {
                in.close();
                clientSocket.close();
                System.out.println("Client connection closed");
            } catch (IOException e) {
                e.printStackTrace();
            }
        }

        private int processNumber(String inputLine) {
            if (!numberPattern.matcher(inputLine).matches()) return -1; //Invalid format, terminate client

            int num;

            try {
                num = Integer.parseInt(inputLine);
            } catch(NumberFormatException e) {  //Invalid format, terminate client
                num = -1;
            }

            return num;
        }
    }


}

LogWriterTask.java:

import java.io.BufferedWriter;
import java.io.IOException;
import java.util.Timer;
import java.util.TimerTask;
import java.util.concurrent.BlockingQueue;

public class LogWriterTask extends Thread {
    private final static int MAX_NINE_DIGIT_RANGE = 1000000000;
    private final static int SUMMARY_WAIT_PERIOD = 10000;
    private int uniqueCount = 0;
    private int duplicateCount = 0;
    private int uniqueTotal = 0;
    private BlockingQueue<Integer> blockingQueue;
    private BufferedWriter bw;
    private int() uniqueNums = new int(MAX_NINE_DIGIT_RANGE);

    public LogWriterTask(BlockingQueue<Integer> blockingQueue, BufferedWriter bw) {
        this.bw = bw;
        this.blockingQueue = blockingQueue;
        Timer timer = new Timer();
        timer.schedule(new SummaryTask(), 0, SUMMARY_WAIT_PERIOD);
    }

    @Override
    public void run() {
        while (true) {
            while (!blockingQueue.isEmpty()) {
                int num = blockingQueue.poll();
                if (uniqueNums(num) == 0) {
                    try {
                        uniqueCount++;
                        uniqueTotal++;
                        bw.write(String.format("%09d", num));
                        bw.newLine();
                        bw.flush();
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                } else {
                    duplicateCount++;
                }

                uniqueNums(num)++;
            }
        }
    }

    class SummaryTask extends TimerTask {
        @Override
        public void run() {
            System.out.printf("Received %d unique numbers, %d duplicates. Unique total: %dn", uniqueCount, duplicateCount, uniqueTotal);
            uniqueCount = 0;
            duplicateCount = 0;
        }
    }
}

Client.java:

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import java.net.Socket;

public class Client {
    private Socket clientSocket;
    private PrintWriter out;
    private BufferedReader in;

    public void startConnection(String ip, int port) {
        try {
            clientSocket = new Socket(ip, port);
            out = new PrintWriter(clientSocket.getOutputStream(), true);
            in = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
        } catch (IOException e) {
            System.err.println("Error when initializing connection");
        }

    }

    public void sendMessage(String msg) {
        try {
            out.println(msg);
            return;
        } catch (Exception e) {
            return;
        }
    }

    public void stopConnection() {
        try {
            in.close();
            out.close();
            clientSocket.close();
        } catch (IOException e) {
            System.err.println("error when closing");
        }

    }
```

sql server – Parameter Sniffing and Multiple Plans in Cache

We have a multi-tenanted database. FirmID is the partition key and we have lots of different firms.

I am running into a parameter sniffing issue and I am having a heck of a time getting around it.

I would rather not use any (Options) on the query.

My latest thought was to change the name of the parameter I am using for the firm. In the snippet below you will see that instead of using @FirmID I called it @Firm611 where 611 is the actual firm of the ID. This will give me a unique query for every firm.

select
    c.ID (_cid),
    c.Name (Name)  
from vwClaims c with(nolock)  
where c.FirmID=@Firm611       
and (c.Name is not null and c.Name!='')
    
select
    c.ID (_cid),
    c.Name (Name)
from vwClaims c with(nolock)
where c.FirmID=@Firm625
and (c.Name is not null and c.Name!='')

After running Brent Ozar’s sp_BlitzCache, I found that it is just compiling down to the same query and causing duplicate cache entries:

Query Plans associated with the same Query Hash

My question is am I reading that result right? Even though I am changing the parameter name, is it really still using the same plan and parameter sniffing?

views – How to apply a sort criteria to multiple displays without erase each individual criteria?

sorry if is a stupid question and for my english, but here I go:

I have a view with multiple displays (don’t ask me why) and I have some sort criterias applied to all the displays and one applied individually to each display. Now I have to add one more sort criteria to all displays but if a I select to apply to all it just erase the individual one. My question is: there is a way to workaround it? Cause it will be a hella burden to apply to each display individually again.

Ps.: I’m just a final user, I don’t have access to the site’s code.

multiple monitors – Change primary mode in xrandr, and delete invalid mode

When I switch to this console, or login, I have to run xrandr:

celiyah@celiyah-pc:~/Code/artifactory/terraform$ xrandr
Screen 0: minimum 320 x 200, current 4880 x 2560, maximum 16384 x 16384
DP-1 connected primary 3440x1440+0+562 (normal left inverted right x axis y axis) 800mm x 335mm
   3440x1440     59.97 +  49.99    29.99  
   2560x1080     60.00  
   1920x1080     60.00    60.00    50.00    59.94  
(...)
   3440x1440_30  29.95* 
HDMI-2 connected 1440x2560+3440+0 left (normal left inverted right x axis y axis) 609mm x 349mm
   2048x1080     24.00  
   1920x1080     60.00    60.00    50.00    59.94    30.00    25.00    24.00    29.97    23.98  
   1920x1080i    60.00    50.00    59.94    50.00  
(...)
   2560x1440_60.00  59.96  
   2560x1440_30.00  29.94* 

…otherwise it chooses that first mode, 3440×1440@60Hz, which doesn’t work for my monitor and results in a lot of churn on the device.

I want to eliminate that mode, but I can’t:

celiyah@celiyah-pc:~/Code/artifactory/terraform$ sudo xrandr --delmode DP-1 3440x1440
X Error of failed request:  BadAccess (attempt to access private resource denied)
  Major opcode of failed request:  140 (RANDR)
  Minor opcode of failed request:  19 (RRDeleteOutputMode)
  Serial number of failed request:  38
  Current serial number in output stream:  39

(same thing happens without sudo)…

How can I prevent it from trying to do this to my poor monitor?

probability or statistics – Binary (binomial) logistic regression with multiple independent variables: calculating the confidence range or prediction

Adding on top to this post here, I am looking for the confidence range (intreval) of the prediction of a multi variant binary logistic regression.

Can someone extend the solution in the post referenced above or provide an alternative?

(*Test data (not full set)*)
testDate = {{449.`, 30.`, 0.`}, {449.`, 30.`, 0.`}, {452.`, 30.`, 
    0.`}, {452.`, 30.`, 0.`}, {562.`, 30.`, 0.`}, {566.`, 30.`, 
    1.`}, {566.`, 30.`, 1.`}, {656.`, 30.`, 1.`}, {656.`, 30.`, 
    1.`}, {658.`, 30.`, 1.`}, {658.`, 30.`, 1.`}, {452.`, 34.`, 
    0.`}, {452.`, 34.`, 0.`}, {452.`, 34.`, 0.`}, {562.`, 34.`, 
    0.`}, {562.`, 34.`, 0.`}, {562.`, 34.`, 0.`}, {658.`, 34.`, 
    0.`}, {658.`, 34.`, 1.`}, {658.`, 34.`, 1.`}, {449.`, 50.`, 
    1.`}, {449.`, 50.`, 1.`}, {449.`, 50.`, 1.`}, {449.`, 50.`, 
    1.`}, {452.`, 50.`, 1.`}, {562.`, 50.`, 1.`}, {562.`, 50.`, 
    1.`}, {566.`, 50.`, 1.`}, {566.`, 50.`, 1.`}, {566.`, 50.`, 
    1.`}, {566.`, 50.`, 1.`}, {656.`, 50.`, 1.`}, {656.`, 50.`, 
    1.`}, {656.`, 50.`, 1.`}, {658.`, 50.`, 1.`}, {658.`, 50.`, 1.`}};
(*Model to fit*)
logit = LogitModelFit(testDate, {P, p, P*p}, {P, p});
Normal(logit)

plugins – Possible way of having multiple Gutenberg instances for a page?

I’m trying to create a system in which the Admin of the wordpress site can create different “versions” of a same page depending on the custom user roles we create for certain users. There’s a fairly good amount of pages that need this feature so my idea is to create a plugin that, for each page created on the backend, can have multiple instances of the gutenberg editor (one per user role) that the page shows different content based on the user role that a certain user has when logged in.

Is there any way I can have some sort of toggle that lets the Admin switch between multiple instances of the Gutenberg editor for any given page?

Any ideas as to where to start looking are appreciated. Thanks!

c++ – Message passing API organization for multiple project executables

We’ve got a project where we have three executables in different subprojects (but underneath completely different repos), lets call them Project A, Project B, and Project C. Project A.exe and Project B.exe have to communicate over the network, they may or may not be on the same machine. We’ve chosen ZMQ to accomplish this. But because these projects are in completely different repos (and for very different purposes, they don’t really share code) we need a common “communication” API, we’ve opted to create a ZMQ library that handles the creation, serialization, and deserialization of different messages, as well as enforcing things like byte alignment. This has allowed us to sync our messages, and has worked great.

But we now have Project C.exe, which doesn’t have anything to do with Project B directly. It also needs to communicate back and forth between with Project A. Our initial thoughts are to simply create a new separate library.

Some of our members have argued against having a separate library for this communication, instead advocating for combining all current and future communication libraries into a single library.

I’m worried about possible compilation bloat and creating artificial dependencies for Project C.exe given it can (and does) exist/get used independently of either of the other libraries, as well as other projects. And beyond the fact that we have an additional library in our project, I don’t really see why we need to combine everything into one library.

What are benefits of using a single “communications” library for this project? Is there an alternative to both options?

python – Merge two dataframes using multiple substring matches

substring_ref = pd.DataFrame({"keyword_1":("Amazon Web Service -", "Virtual Apps"),  "keyword_2": ("","Service")})

sales = pd.DataFrame({"product_name":("Amazon Web Service - Automation", "Web", "Citrix Virtual Apps Premium OnPrem", "Citrix Virtual Apps Premium Service"), "Value":(10, 90, 500, 340)})

I have two dataframes that I want to merge based on either keyword_1 (if only one keyword is present) OR keyword_1 and keyword_2 both being present in the product_name field of the sales dataframe.

The end result should be this:

keyword_1 keyword_2 product_name Value
Amazon Web Service – Amazon Web Service – Automation 10
Web 90
Citrix Virtual Apps Premium OnPrem 500
Virtual Apps Service Citrix Virtual Apps Premium Service 340