apache kafka – Efficient options for stream joins (with big time gaps)

Use Case: I need to connect two stream sources (eg orders(order_id, order_val) and shipments(shipment_id, shipment_val)) based on an ID (order_id = shipment_id) and generate a new event shipment_order(id, order_val, shipment_val)

Note that the gap between events can be very large (1 year – 2 years).
Ex: order(order_id = 1, value=1) could arrive today, but the shipment(shipment_id=1, value=2) could arrive after one year.

I examine patterns to achieve window-efficient stream joins:

  1. Save events to DynamoDB (or other datastore) and reissue events via DynamoDB streams (modify data collection in general). If either a DDB order event or a DDB shipping event arrives, I will check DynamoDB for both events using the DDB event ID and make a join and send a new cargo_order

  2. Kafka Stream is used with large windows (2 years)


  1. What are some good patterns for getting such big window joins?

  2. Is stateful systems such as Kafka recommending a 2-year window join? (What are the implications?)

Note: Assume that the system handles about 200 million events per day with spikes. By efficiency I mean the overall time / cost efficiency (and tradeoffs).

Apache – Apache2 does not run on High Sierra without a homebrew

This is a post I found on this topic:
Apache2 (Httpd) does not work anymore after updating to High Sierra

But I have a situation where httpd is not running
After enabling Preferences -> Sharing -> Web sharing

The relevant factors:

I bought this machine with Sierra installed.

I updated it on High Sierra.

I tried to set localhost as my homepage in my browsers.

The browser then indicated that it can not connect to localhost.

I googled a bit and found the post with these references.

I have not installed homebrew on this computer.

I have never used homebrew.

But there is a possibility that homebrew was
At some point installed and upgrades to the system have not reversed it.
I find in / etc / apache2 / a httpd.conf ~ previous
and many files in ./extra, where ~ previous is attached to different files.

The question is: How can I check if Homebrew has been installed and if I can safely run the found uninstall script?

There is a related problem with Ethernet interfaces:
If a browser is used to contact localhost and an error occurs that says it can not connect to localhost, I've found that I no longer have access
the internet (App Store does not connect if I do not have internet connection).
BUT I can access file sharing machines on the local network. So I know that the interface works.

I would like to clarify that. I do
Development and want to work localhost. I like
localhost as Landing Pad Homepage for
my browser.

Thank you for your time and attention

redirect – Apache redirectmatch collection groups are not properly routed to the destination

I'm trying to redirect requests to my old Git server (with my old username) to the new server (with my new username).

I try to set the following conditions:

  1. If the URL is http (s): //git.glmdev.tech/glmdev/any/url/here, forward it to https://git.garrettmills.dev/garrettmills/any/url/here.

  2. If the URL is http (s): //git.glmdev.tech/glmdev, redirect it to https://git.glmdev.tech/garrettmills

  3. Otherwise forward all http (s): //git.glmdev.tech/any/other/url to https://git.garrettmills.dev/any/other/url

I am trying to use Apache2 RedirectMatch to do this using collection groups. I have the following virtual host configuration:

    ServerName git.glmdev.tech
    RedirectMatch 301 /glmdev/(.*) https://git.garrettmills.dev/garrettmills/$1
    Redirect /glmdev/ https://git.garrettmills.dev/garrettmills/
    Redirect / https://git.garrettmills.dev/

    ServerName git.glmdev.tech
    SSLEngine on
    SSLCertificateFile /etc/glmdev/ssl/_star/cert.crt
    SSLCertificateKeyFile /etc/glmdev/ssl/_star/cert.key
    SSLCertificateChainFile /etc/glmdev/ssl/_star/ca.crt
    RedirectMatch 301 /glmdev/(.*) https://git.garrettmills.dev/garrettmills/$1
    Redirect /glmdev/ https://git.garrettmills.dev/garrettmills/
    Redirect / https://git.garrettmills.dev/

Conditions 2 and 3 work as expected, but navigate to some http (s): //git.glmdev.tech/glmdev/repositoryname and redirect to https://git.garrettmills.dev/garrettmills

Not sure why the collection group is not found or appended to the URL. Every help is appreciated. Thank you in advance.

virtualhost – The default Apache page opens instead of my webpage

This is my virtual host setting. When I try to load my website, the default Apache page is displayed. After a few updates, however, my website will load. I tried to look for this problem, but everyone is talking about configuring the virtual host. can someone find out what's wrong in my virtual host configuration? I also disable the default configuration of the virtual host 000-default.com.conf.

    ServerAdmin abc@example.com
    ServerName test.example.com
    ServerAlias www.test.example.com
    DocumentRoot /var/www/apps/staging/example
    ProxyRequests off
            Order deny,allow
            Allow from all
            ProxyPass http://localhost:4204/
            ProxyPassReverse http://localhost:4204/

Virtualmin / Webmin WordPress Multisite Apache Config

Hopefully this is the place to ask these questions. If not, please forward me to the right place.

I have inherited a server with CentOS 7.6 and Webmin 1.930 / Virtualmin 6.07 Pro.

I managed to set up a WordPress multisite (5.2.3). The sites are subdomain.domain.com/site1, subdomain.domain.com/site2 … and I want each site to have its own domain. I think I have to set this up in the Apache configuration and want to make sure that Wemnin / Virtualmin does not override this.

Part of the solution may be to move all multisites to different subdomains of the main domain and then map each domain I want to subdomains. However, this would require a bit more DNS setup and would not be sure how to do this in Apache.

Googled a lot but no luck, how do I set that?

Greetings, Ben

I also want to use Lets Encrypt for these domains.

virtualhost – An error occurred when Apache was configured to serve the virtual host in the local domain on the LAN

I'm using Apache 2.4.29 on Ubuntu 18. With a default domain, Apache worked fine, but I'm trying to set up additional subdomains. This is my virtual host configuration:

adam@gondolin:~$ cat /etc/apache2/sites-available/edoras_mysql.conf

    ServerName edoras_mysql.gondolin.localdomain
    ServerAdmin webmaster@localhost
    DocumentRoot "/var/www/edoras_mysql"

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined

The default setting for conf is unchanged.

I ran a2ensite edoras_mysql and it did not complain and restart the Apache daemon, but browsing the site leads to:

We can’t connect to the server at edoras_mysql.gondolin.localdomain.

from the same machine. surfing gondolin.localdomain although works.

This is my /etc/hosts:

adam@gondolin:~$ cat /etc/hosts   localhost localhost.localdomain gondolin   gondolin     gondolin.localdomain gondolin

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

If I did something fundamentally wrong, I do not see it. I should have that after everything I read on the net.

I use that gondolin Host as a DHCP and DNS server and gateway with iptables, which works fine so I do not see any problems with that.

kerberos – Urgent: Apache Kudu error with IllegalStateException: security issue

Urgent help requested.
I'm trying to run my code on Apache Kudu. While it works well on my local kudu setup where I have the master and tablet server on the same computer. Now with the manual installation of Kudu via RPM package installation on Cloudera Hadoop env, I get under exception.

I try to load one csv File over sparks and persistently on Kudu.
I have already set up the following in a working state:

  1. Installed Kudu Master and Tablet on different nodes (and they run)
  2. A Kereberos ticket for the new user (kudu) is set up on all cluster nodes. (With only one master and one tablet, however, it is only necessary to be related to two of them.)
  3. The Ranger guidelines are available for each component, such as sparks, hive, etc., to transfer or transfer a job to another component. The exception is as follows

    19/09/10 18:33:43 INFO FileSourceScanExec: Schedule a scan with recycle bin packing, maximum size: 4194304 bytes, open cost is considered scanning 4194304 bytes.
    19/09/10 18:33:43 INFO CoreUtils $: Elapsed time (KuduLoader load csv-> df): 1.727817664s
    19/09/10 18:33:43 INFO KuduLoader $: kuduMaster: kudu: //mynode.ic.company.net: 7051
    19/09/10 18:33:43 INFO KuduContext: Login as Principal kudu@mycluster.IC.COMPANY.NET with the key combination /home/kudu/kudu.keytab
    Exception in thread "main" java.lang.IllegalStateException
    at org.apache.kudu.shaded.com.google.common.base.Preconditions.checkState (Preconditions.java:494)
    at org.apache.kudu.shaded.com.google.common.net.HostAndPort.getPort (HostAndPort.java:110)
    at org.apache.kudu.util.NetUtil.parseString (NetUtil.java:68)
    at org.apache.kudu.util.NetUtil.parseStrings (NetUtil.java:85)
    at org.apache.kudu.client.AsyncKuduClient $ AsyncKuduClientBuilder. (AsyncKuduClient.java:2372)
    at org.apache.kudu.spark.kudu.KuduClientCache $ .getAsyncClient (KuduContext.scala: 525)
    at org.apache.kudu.spark.kudu.KuduContext.asyncClient $ lzycompute (KuduContext.scala: 136)
    at org.apache.kudu.spark.kudu.KuduContext.asyncClient (KuduContext.scala: 135)
    at org.apache.kudu.spark.kudu.KuduContext.syncClient $ lzycompute (KuduContext.scala: 133)
    at org.apache.kudu.spark.kudu.KuduContext.syncClient (KuduContext.scala: 133)
    at org.apache.kudu.spark.kudu.KuduContext $$ anon $ 1.run (KuduContext.scala: 147)
    at org.apache.kudu.spark.kudu.KuduContext $$ anon $ 1.run (KuduContext.scala: 145)
    at java.security.AccessController.doPrivileged (native method)
    at javax.security.auth.Subject.doAs (Subject.java:360)
    at org.apache.kudu.spark.kudu.KuduContext. (KuduContext.scala: 145)
    at org.apache.kudu.spark.kudu.KuduContext. (KuduContext.scala: 67)
    at com.company.new.project.kuduloader.KuduLoader.store (KuduLoader.scala: 86)
    at com.company.new.project.kuduloader.Application $$ anonfun $ main $ 1 $$ anonfun $ mcV $ sp $ Apply 1.mcV $ sp (Application.scala: 42)
    at com.company.new.project.kuduloader.Application $$ anonfun $ main $ 1 $$ anonfun $ mcV $ sp $ Apply (Application.scala: 42)
    at com.company.new.project.kuduloader.Application $$ anonfun $ main $ 1 $$ anonfun $ mcV $ sp $ Apply (Application.scala: 42)
    at com.company.new.project.kuduloader.CoreUtils $ .time (CoreUtils.scala: 8)
    at com.company.new.project.kuduloader.Application $$ anonfun $ main $ 1.apply $ mcV $ sp (Application.scala: 41)
    at com.company.new.project.kuduloader.Application $$ anonfun $ main $ 1.apply (Application.scala: 27)
    at com.company.new.project.kuduloader.Application $$ anonfun $ main $ 1.apply (Application.scala: 27)
    at com.company.new.project.kuduloader.CoreUtils $ .time (CoreUtils.scala: 8)
    at com.company.new.project.kuduloader.Application $ .main (Application.scala: 27)
    at com.company.new.project.kuduloader.Application.main (Application.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (native method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start (SparkApplication.scala: 52)
    at org.apache.spark.deploy.SparkSubmit $ .org $ apache $ spark $ deploy $ SparkSubmit $$ runMain (SparkSubmit.scala: 904)
    at org.apache.spark.deploy.SparkSubmit $ .doRunMain $ 1 (SparkSubmit.scala: 198)
    at org.apache.spark.deploy.SparkSubmit $ .submit (SparkSubmit.scala: 228)
    at org.apache.spark.deploy.SparkSubmit $ .main (SparkSubmit.scala: 137)
    at org.apache.spark.deploy.SparkSubmit.main (SparkSubmit.scala)