Tag: large

mysql – Monitoring slow queries on large database (Percona 5.6)

I am looking for a solution to monitor very large production database (many replicas, over 200 tables, some of tables have 10-50 GB of data) with a lot of traffic (a wide set of query complexity).

The main goal is to keep an eye on heavy queries after application code change. We have experienced that new code introduced non-optimal sql queries.

The database dump is very huge so we use a sample on staging environments only. It is almost impossible to detect performance issue on staging in this case.

I thought about using something like New Relic but for such big infrastructure costs are too high and I am not sure it affects performance in a negative way.
On the other I can use MySQL slow log. I enabled it on some replicas with a log rate limit. After one month I do not see I/O speed impact but I am not sure about enabling it for all replicas and the master database.
I addition I found this article http://blog.symedia.pl/2016/10/performance-impact-general-slow-query-log.html and it says something different that I read in the past.

What would you do to monitor large database?

mysql – Does large text datatypes reserves storage on disk?

I’m design an DB for an app, in a table there is a column MEDIUMTEXT which according to mysql doc can storage on disk upto 16MB.

So my question is if I’m storing a text of 1Mb in this column, will it still reserves 16MB storage/block on disk for this particular record.

And also while reading all columns of multiple rows at once i.e 100 rows, what impact will have on memory and CPU usage for reading such large size storage columns?

performance tuning – Avoiding CopyTensor of large array in Compiled functions

After reading about how to avoid the CopyTensor in various cases I still do not understand the optimal answer for the Fast-Hadamard transform below (CopyTensor appears at line 4 of the code):

 FHT = Compile({{vec, _Complex, 1}}, 
  Module({i, j, x, y, h, state, len, num},
   h = 1;
   state = vec;
   len = Length(state);
   num = (Sqrt(2)^Log(2, len));
   While(h < len,
    For(i = 1, i <= len - 1, i = i + 2*h,
     For (j = i, j <= i + h - 1, j++,
       x = state((j));
       y = state((j + h));
       state((j)) = x + y;
       state((j + h)) = x - y;
       );
     );
    h *= 2;
    );
   state/num
   ), RuntimeOptions -> {"CatchMachineUnderflow" -> False, 
    "CatchMachineOverflow" -> False, 
    "CatchMachineIntegerOverflow" -> False, 
    "CompareWithTolerance" -> False, "EvaluateSymbolically" -> False, 
    "RuntimeErrorHandler" -> False, "WarningMessages" -> False}, 
  CompilationOptions -> {"ExpressionOptimization" -> True, 
    "InlineExternalDefinitions" -> True}, "CompilationTarget" -> "C", 
  RuntimeAttributes -> {Listable}, Parallelization -> True
  )

I know that mathematica has a pre-compiled function but it is much slower than my example. The only problem that I have is that it is not clear how to pass the array by reference. Is there an easy efficient answer to that? I am interested in transforming arrays of $2^{24}-2^{30}$ elements.

Is there a better way to unit test a repository for a large aggregate root?

In our project we use domain-driven design and our customer aggregate root is large. The unit tests for the aggregate root itself are simple (~50 or so tests), but I am struggling with the repository tests. Other aggregate roots in our system are not this large (number of fields), and the database table for this data is not normalized (around ~40-70 columns). The design of this table was done about 10 years ago and normalizing this table is not feasible right now.

Normally I would add a test for each separate field because I find that it provides better feedback when a bug occurs, code has changed, and the tests fail. It’s really easy to find that test and fix that specific bug. In this case I have avoided that because of the number of fields and amount of setup code, and grouped the assertions together by the operation or scenario moreso than a unit.

What is not shown in this test either is that for Add there are still ~20-30 other columns in this table that must have default values, but I have chosen to not verify those due to test value assuming the database will default them. I’m not sure I am okay with that outcome because the database is not normalized. If it were, where some columns were on join tables or separate tables it would not be an issue at all.

How have you dealt with unit testing large aggregate roots like this? I thought about separating the factory for constructing Customer and unit testing that with tests for each field, and creating another factory or builder with the same intent for constructing Data.Customer. If I take that approach, the unit tests for the repository are straight forward given that they become just identity based verifications like context.Verify(...Add...) and “`builder.AddCustomerInfo(…)`.

Some of the code has ... values that replace the actual values in our test code.

Repository

namespace OurProduct.Core.Infrastructure
{
    using System;
    using System.Collections.Generic;
    using System.Data.Entity;
    using System.Linq;
    using Business;
    using Data.Infrastructure;

    /// <inheritdoc cref="ICustomerRepository" />
    public class CustomerRepository : ICustomerRepository
    {
        private readonly IDateTime time;

        /// <summary>
        /// Initializes a new instance of the <see cref="CustomerRepository"/> class.
        /// </summary>
        /// <param name="database">The database context that will be used to access data.</param>
        /// <param name="time">A time object that will be used to get system clock info.</param>
        public CustomerRepository(IDataContext database, IDateTime time)
        {
            this.time = time;

            Database = database;
            Query    = Database.Customers
                               .Include(p => p.Branch)
                               .Include(p => p.Branch.Bank)
                               .Include(p => p.CreditClassification)
                               .Include(p => p.LoanOfficer)
                               .Include(p => p.CustomerStatus)
                               .Include(p => p.CustomerType);
        }

        /// <summary>
        /// Gets the database context that will be used to access data.
        /// </summary>
        public IDataContext Database
        {
            get;
        }

        /// <summary>
        /// Gets the queryable expression that will be used to load customers.
        /// </summary>
        public IQueryable<Data.Customer> Query
        {
            get;
        }

        /// <inheritdoc />
        public void Add(Customer customer)
        {
            var entity = new Data.Customer
                         {
                             CustomerId         = customer.Id,
                             CustomerNumber     = customer.Number,
                             CustomerFirstName  = customer.FirstName,
                             CustomerMiddleName = customer.MiddleName,
                             CustomerLastName   = customer.LastName,
                             CustomerName       = customer.FullName,
                             BusinessName       = customer.BusinessName, 
                             TaxId              = customer.TaxId,
                             PhoneFax           = customer.ContactInfo.FaxNumber,
                             PhoneHome          = customer.ContactInfo.HomeNumber,
                             PhoneMobile        = customer.ContactInfo.MobileNumber,
                             PhoneWork          = customer.ContactInfo.WorkNumber, 
                             Email              = customer.ContactInfo.Email,
                             Address1           = customer.ContactInfo.Address.Address1,
                             Address2           = customer.ContactInfo.Address.Address2,
                             City               = customer.ContactInfo.Address.City,
                             State              = customer.ContactInfo.Address.State,
                             ZipCode            = customer.ContactInfo.Address.Zip,
                             ...
                             ModifiedDate       = time.Now,
                             BankId             = ((ICustomer) customer).BankId,
                             CustomerBranchId   = ((ICustomer) customer).BranchId,
                             CustomerTypeId     = ((ICustomer) customer).TypeId,
                             CustomerStatusId   = ((ICustomer) customer).StatusId,
                             ClassificationId   = ((ICustomer) customer).RiskId,
                             CustomerOfficerId  = ((ICustomer) customer).OfficerId, 
                         };

            Database.Add(entity);
        }

        /// <inheritdoc />
        public IEnumerable<Customer> Find(IEnumerable<string> searchTerms, int? skip = 0, int? take = 25)
        {
            throw new NotImplementedException(@"Implement using full-text search in SQL using an indexed view.");
        }

        /// <inheritdoc />
        public Customer FindById(Guid id)
        {
            Data.Customer entity;

            try
            {
                entity = Query.Single(p => p.CustomerId == id);
            }
            catch (InvalidOperationException ex)
            {
                throw new ArgumentException(@"Identity does not exist", nameof(id), ex);
            }

            Customer customer = CustomerFactory.Create(entity);

            return customer;
        }

        /// <inheritdoc />
        public void Update(Customer customer)
        {
            var entity = Database.Find<Data.Customer>(customer.Id);

            if (entity == null)
            {
                throw new ArgumentException(@"The specified customer does not exist in the repository.", 
                                            nameof(customer));
            }

            entity.CustomerFirstName  = customer.FirstName;
            entity.CustomerMiddleName = customer.MiddleName;
            entity.CustomerLastName   = customer.LastName;
            entity.CustomerNumber     = customer.FullName;
            entity.BusinessName       = customer.BusinessName;
            entity.TaxId              = customer.TaxId;
            entity.PhoneFax           = customer.ContactInfo.FaxNumber;
            entity.PhoneHome          = customer.ContactInfo.HomeNumber;
            entity.PhoneMobile        = customer.ContactInfo.MobileNumber;
            entity.PhoneWork          = customer.ContactInfo.WorkNumber;
            entity.Email              = customer.ContactInfo.Email;
            entity.Address1           = customer.ContactInfo.Address.Address1;
            entity.Address2           = customer.ContactInfo.Address.Address2;
            entity.City               = customer.ContactInfo.Address.City;
            entity.State              = customer.ContactInfo.Address.State;
            entity.ZipCode            = customer.ContactInfo.Address.Zip;
            entity.ModifiedDate       = time.Now;
            entity.BankId             = ((ICustomer) customer).BankId;
            entity.CustomerBranchId   = ((ICustomer) customer).BranchId;
            entity.CustomerTypeId     = ((ICustomer) customer).TypeId;
            entity.CustomerStatusId   = ((ICustomer) customer).StatusId;
            entity.ClassificationId   = ((ICustomer) customer).RiskId;
            entity.CustomerOfficerId  = ((ICustomer) customer).OfficerId;
        }

        /// <summary>
        /// Factory class that creates new <see cref="Customer"/> instances.
        /// </summary>
        private static class CustomerFactory
        {
            /// <summary>
            /// Creates a new customer from the specified data entity.
            /// </summary>
            /// <param name="entity">The <see cref="Data.Customer"/> data entity containing the data that will be used to create a new customer.</param>
            /// <returns>Returns a new <see cref="Customer"/> instance.</returns>
            public static Customer Create(Data.Customer entity)
            {
                var bank    = new Bank(entity.Branch.Bank.BankId, entity.Branch.Bank.BankName);
                var region  = new Region(entity.Branch.RegionId, entity.Branch.Region.RegionName);
                var branch  = new Branch(entity.Branch.BranchId, entity.Branch.BranchName, bank, region);
                var risk    = new CustomerRisk(entity.CreditClassification.ClassificationId, entity.CreditClassification.ClassificationName);
                var officer = new Officer(entity.LoanOfficer.OfficerId, entity.LoanOfficer.OfficerName, entity.LoanOfficer.OfficerTitle);
                var status  = new CustomerStatus(entity.CustomerStatus.CustomerStatusId, entity.CustomerStatus.CustomerStatusDescription, entity.CustomerStatus.IsActive);
                var type    = new CustomerType(entity.CustomerType.CustomerTypeId, entity.CustomerType.CustomerTypeDescription);

                var customer = new Customer(entity.CustomerId,
                                            entity.CustomerName,
                                            entity.CustomerNumber,
                                            type,
                                            branch,
                                            officer,
                                            status,
                                            risk)
                               {
                                   FirstName    = entity.CustomerFirstName,
                                   MiddleName   = entity.CustomerMiddleName,
                                   LastName     = entity.CustomerLastName,
                                   BusinessName = entity.BusinessName,
                                   TaxId        = entity.TaxId
                               };

                customer.ContactInfo.Email            = entity.Email;
                customer.ContactInfo.Address.Address1 = entity.Address1;
                customer.ContactInfo.Address.Address2 = entity.Address2;
                customer.ContactInfo.Address.City     = entity.City;
                customer.ContactInfo.Address.State    = entity.State;
                customer.ContactInfo.Address.Zip      = entity.ZipCode;
                customer.ContactInfo.FaxNumber        = entity.PhoneFax;
                customer.ContactInfo.HomeNumber       = entity.PhoneHome;
                customer.ContactInfo.WorkNumber       = entity.PhoneWork;
                customer.ContactInfo.MobileNumber     = entity.PhoneMobile;

                return customer;
            }
        }
    }
}

Unit Tests

namespace OurProduct.Core.Tests
{
    using System;
    using System.Linq;
    using Business;
    using Data;
    using Data.Infrastructure;
    using Infrastructure;
    using Microsoft.VisualStudio.TestTools.UnitTesting;
    using Moq;
    using Bank = Business.Bank;
    using Branch = Business.Branch;
    using Customer = Business.Customer;
    using CustomerStatus = Business.CustomerStatus;
    using CustomerType = Business.CustomerType;
    using Region = Business.Region;

    (TestClass)
    public class CustomerRepositoryTests
    {
        (TestMethod)
        public void Add_CustomerIsNotNull_AddsCustomerToContext()
        {
            Guid customerId = Guid.Parse("{341E4B1C-EBA3-4587-B7A6-3C9641C39191}");
            Guid bankId     = Guid.Parse("{BE3590B5-F570-44B6-BAF1-86597CC404A6}");
            Guid branchId   = Guid.Parse("{2CA73ABE-0EFC-4C8F-9BD9-8D1F4E0E74DE}");
            Guid regionId   = Guid.Parse("{3C4D243F-47AF-4224-B805-55E2D8EB0DE0}");
            Guid riskId     = Guid.Parse("{DC0A5EF1-2376-4FA9-B09A-69CD1B28CEF5}");
            Guid officerId  = Guid.Parse("{8F02B1F8-1551-4124-B9B7-492FD18830DD}");
            Guid statusId   = Guid.Parse("{C53BE489-DDC5-4178-BA83-3422A58CC1D8}");
            Guid typeId     = Guid.Parse("{F2118364-64FB-44DC-8AC2-19223447A125}");

            var bank     = new Bank(bankId, "...");
            var region   = new Region(regionId, "West US");
            var branch   = new Branch(branchId, "...", bank, region);
            var risk     = new CustomerRisk(riskId, "Low Risk");
            var officer  = new Officer(officerId, "Bob Smith", "Senior Lender");
            var status   = new CustomerStatus(statusId, "Active", true);
            var type     = new CustomerType(typeId, "Consumer");

            var customer = new Customer(customerId,
                                        "John J Doe",
                                        "CC001",
                                        type,
                                        branch,
                                        officer,
                                        status,
                                        risk)
                           {
                               FirstName    = "John",
                               MiddleName   = "J",
                               LastName     = "Doe",
                               BusinessName = "Local Shipping, Inc.",
                               TaxId        = "123456789"
                           };

            customer.ContactInfo.Email            = "jjdoe@....net";
            customer.ContactInfo.Address.Address1 = "...";
            customer.ContactInfo.Address.Address2 = "";
            customer.ContactInfo.Address.City     = "...";
            customer.ContactInfo.Address.State    = "..";
            customer.ContactInfo.Address.Zip      = "...";
            customer.ContactInfo.FaxNumber        = "123 456 7890";
            customer.ContactInfo.HomeNumber       = "123 456 7891";
            customer.ContactInfo.WorkNumber       = "123 456 7892";
            customer.ContactInfo.MobileNumber     = "123 456 7893";

            var context    = new Mock<IDataContext>();
            var time       = new Mock<IDateTime>();
            var repository = new CustomerRepository(context.Object, time.Object);

            repository.Add(customer);

            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.CustomerId == customer.Id)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.BankId == ((ICustomer)customer).BankId)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.CustomerBranchId == ((ICustomer)customer).BranchId)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.CustomerTypeId == ((ICustomer)customer).TypeId)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.CustomerStatusId == ((ICustomer)customer).StatusId)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.ClassificationId == ((ICustomer)customer).RiskId)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.CustomerNumber == customer.Number)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.CustomerName == customer.FullName)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.CustomerFirstName == customer.FirstName)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.CustomerMiddleName == customer.MiddleName)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.CustomerLastName == customer.LastName)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.BusinessName == customer.BusinessName)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.TaxId == customer.TaxId)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.Email == customer.ContactInfo.Email)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.Address1 == customer.ContactInfo.Address.Address1)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.Address2 == customer.ContactInfo.Address.Address2)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.City == customer.ContactInfo.Address.City)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.State == customer.ContactInfo.Address.State)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.ZipCode == customer.ContactInfo.Address.Zip)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.PhoneHome == customer.ContactInfo.HomeNumber)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.PhoneFax == customer.ContactInfo.FaxNumber)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.PhoneWork == customer.ContactInfo.WorkNumber)), Times.Once);
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.PhoneMobile == customer.ContactInfo.MobileNumber)), Times.Once);
            ...
            context.Verify(p => p.Add(It.Is<Data.Customer>(q => q.ModifiedDate == time.Object.Now)), Times.Once);
        }

        (TestMethod)
        public void FindById_IdentityExists_ReturnsCustomer()
        {
            var entity = new Data.Customer
            {
                CustomerId           = Guid.Parse("{E9B6F735-7984-4746-92CF-3EEA9E742209}"),
                CustomerNumber       = "CC001",
                CustomerName         = "John J Smith",
                CustomerFirstName    = "John",
                CustomerMiddleName   = "J",
                CustomerLastName     = "Smith",
                Email                = "johnjsmith@....net",
                Address1             = "...",
                Address2             = "",
                City                 = "...",
                State                = "..",
                ZipCode              = "...", 
                PhoneFax             = "123 456 7890",
                PhoneWork            = "123 456 7891",
                PhoneMobile          = "123 456 7892",
                PhoneHome            = "123 456 7893", 
                BusinessName         = "Local Shipping, Inc.",
                TaxId                = "123456789",
                CreditClassification = new CreditClassification
                                       {
                                           ClassificationId   = Guid.Parse("{C31C1C79-5181-4D06-86EA-7AEDECDB8A01}"),
                                           ClassificationName = "Low Risk"
                                       },
                CustomerType         = new Data.CustomerType
                                       {
                                           CustomerTypeId          = Guid.Parse("{3C1236F0-145D-4258-AC54-3EF1331B0B66}"),
                                           CustomerTypeDescription = "Consumer"
                                       },
                CustomerStatus       = new Data.CustomerStatus
                                       {
                                           CustomerStatusId          = Guid.Parse("{0567B8ED-F947-4BF5-936F-630AB3677095}"),
                                           CustomerStatusDescription = "Active"
                                       },
                LoanOfficer          = new LoanOfficer
                                       {
                                           OfficerId    = Guid.Parse("{58DEB0AD-3830-437E-B22D-04376AB5296C}"),
                                           OfficerName  = "Bob Doe",
                                           OfficerTitle = "Senior Lender"
                                       },
                Branch               = new Data.Branch
                                       {
                                           BranchId   = Guid.Parse("{003ED79C-938E-4DB8-9A3C-2AA35E64DAEB}"),
                                           BranchName = "...",
                                           Region     = new Data.Region
                                                        {
                                                            RegionId   = Guid.Parse("{F2992F17-B94A-4CE5-83D1-B3E6A2AC31A7}"),
                                                            RegionName = "West US"
                                                        },
                                           Bank       = new Data.Bank
                                                        {
                                                            BankId   = Guid.Parse("{11D291A4-3F93-443B-93C8-88621F25891A}"),
                                                            BankName = "..."
                                                        }
                                       }
            };

            var context = new Mock<IDataContext>();

            context.Setup(p => p.Customers).Returns(new() {entity}.AsQueryable);

            var repository = new CustomerRepository(context.Object, Mock.Of<IDateTime>());

            Customer customer = repository.FindById(entity.CustomerId);

            Assert.AreEqual("{e9b6f735-7984-4746-92cf-3eea9e742209}", customer.Id.ToString("B"));
            Assert.AreEqual("{11d291a4-3f93-443b-93c8-88621f25891a}", ((ICustomer)customer).BankId.ToString("B"));
            Assert.AreEqual("{003ed79c-938e-4db8-9a3c-2aa35e64daeb}", ((ICustomer)customer).BranchId.ToString("B"));
            Assert.AreEqual("{3c1236f0-145d-4258-ac54-3ef1331b0b66}", ((ICustomer)customer).TypeId.ToString("B"));
            Assert.AreEqual("{0567b8ed-f947-4bf5-936f-630ab3677095}", ((ICustomer)customer).StatusId.ToString("B"));
            Assert.AreEqual("{c31c1c79-5181-4d06-86ea-7aedecdb8a01}", ((ICustomer)customer).RiskId.ToString("B"));
            Assert.AreEqual("CC001", customer.Number);
            Assert.AreEqual("John J Smith", customer.FullName);
            Assert.AreEqual("John", customer.FirstName);
            Assert.AreEqual("J", customer.MiddleName);
            Assert.AreEqual("Smith", customer.LastName);
            Assert.AreEqual("johnjsmith@....net", customer.ContactInfo.Email);
            Assert.AreEqual("...", customer.ContactInfo.Address.Address1);
            Assert.AreEqual("", customer.ContactInfo.Address.Address2);
            Assert.AreEqual("...", customer.ContactInfo.Address.City);
            Assert.AreEqual("...", customer.ContactInfo.Address.State);
            Assert.AreEqual("81003", customer.ContactInfo.Address.Zip);
            Assert.AreEqual("123 456 7890", customer.ContactInfo.FaxNumber);
            Assert.AreEqual("123 456 7893", customer.ContactInfo.HomeNumber);
            Assert.AreEqual("123 456 7892", customer.ContactInfo.MobileNumber);
            Assert.AreEqual("123 456 7891", customer.ContactInfo.WorkNumber);
            Assert.AreEqual("Low Risk", customer.Risk);
            Assert.AreEqual("Consumer", customer.Type);
            Assert.AreEqual("Active", customer.Status);
            Assert.AreEqual("...", customer.Branch);
            Assert.AreEqual("...", customer.Bank);
            Assert.AreEqual("Bob Doe", customer.Officer);
            Assert.AreEqual("Local Shipping, Inc.", customer.BusinessName);
            Assert.AreEqual("123456789", customer.TaxId);
        }

        (TestMethod)
        public void FindById_IdentityDoesNotExist_ThrowsException()
        {
            var repository = new CustomerRepository(Mock.Of<IDataContext>(), Mock.Of<IDateTime>());

            Assert.ThrowsException<ArgumentException>(() => repository.FindById(Guid.Empty));
        }

        (TestMethod)
        public void Update_IdentityDoesNotExist_ThrowsException()
        {
            var customer = new Customer(Guid.Empty, 
                                        "John J Doe",
                                        "CC001",
                                        Mock.Of<ICustomerType>(),
                                        Mock.Of<IBranch>(),
                                        Mock.Of<IOfficer>(),
                                        Mock.Of<ICustomerStatus>(),
                                        Mock.Of<ICustomerRisk>());

            var repository = new CustomerRepository(Mock.Of<IDataContext>(), Mock.Of<IDateTime>());

            Assert.ThrowsException<ArgumentException>(() => repository.Update(customer));
        }

        (TestMethod)
        public void Update_IdentityExists_UpdatesCustomer()
        {
            Guid customerId = Guid.Parse("{341E4B1C-EBA3-4587-B7A6-3C9641C39191}");
            Guid bankId     = Guid.Parse("{BE3590B5-F570-44B6-BAF1-86597CC404A6}");
            Guid branchId   = Guid.Parse("{2CA73ABE-0EFC-4C8F-9BD9-8D1F4E0E74DE}");
            Guid regionId   = Guid.Parse("{3C4D243F-47AF-4224-B805-55E2D8EB0DE0}");
            Guid riskId     = Guid.Parse("{DC0A5EF1-2376-4FA9-B09A-69CD1B28CEF5}");
            Guid officerId  = Guid.Parse("{8F02B1F8-1551-4124-B9B7-492FD18830DD}");
            Guid statusId   = Guid.Parse("{C53BE489-DDC5-4178-BA83-3422A58CC1D8}");
            Guid typeId     = Guid.Parse("{F2118364-64FB-44DC-8AC2-19223447A125}");

            var bank     = new Bank(bankId, "...");
            var region   = new Region(regionId, "West US");
            var branch   = new Branch(branchId, "...", bank, region);
            var risk     = new CustomerRisk(riskId, "Low Risk");
            var officer  = new Officer(officerId, "Bob Smith", "Senior Lender");
            var status   = new CustomerStatus(statusId, "Active", true);
            var type     = new CustomerType(typeId, "Consumer");

            var customer = new Customer(customerId,
                                        "John J Doe",
                                        "CC001",
                                        type,
                                        branch,
                                        officer,
                                        status,
                                        risk)
                           {
                               FirstName    = "John",
                               MiddleName   = "J",
                               LastName     = "Doe",
                               BusinessName = "Local Shipping, Inc.",
                               TaxId        = "123456789"
                           };

            customer.ContactInfo.Email            = "jjdoe@....net";
            customer.ContactInfo.Address.Address1 = "...";
            customer.ContactInfo.Address.Address2 = "";
            customer.ContactInfo.Address.City     = "...";
            customer.ContactInfo.Address.State    = "...";
            customer.ContactInfo.Address.Zip      = "...";
            customer.ContactInfo.FaxNumber        = "123 456 7890";
            customer.ContactInfo.HomeNumber       = "123 456 7891";
            customer.ContactInfo.WorkNumber       = "123 456 7892";
            customer.ContactInfo.MobileNumber     = "123 456 7893";

            var context = new Mock<IDataContext>();
            var entity  = new Data.Customer {CustomerId = customer.Id};

            context.Setup(p => p.Find<Data.Customer>(It.Is<Guid>(q => q == customer.Id)))
                   .Returns(entity);

            var repository = new CustomerRepository(context.Object, Mock.Of<IDateTime>());

            repository.Update(customer);

            Assert.AreEqual(customer.FirstName, entity.CustomerFirstName);
            Assert.AreEqual(customer.MiddleName, entity.CustomerMiddleName);
            Assert.AreEqual(customer.LastName, entity.CustomerLastName);
            Assert.AreEqual(customer.FullName, entity.CustomerNumber);
            Assert.AreEqual(customer.BusinessName, entity.BusinessName);
            Assert.AreEqual(customer.TaxId, entity.TaxId);
            Assert.AreEqual(customer.ContactInfo.FaxNumber, entity.PhoneFax);
            Assert.AreEqual(customer.ContactInfo.HomeNumber, entity.PhoneHome);
            Assert.AreEqual(customer.ContactInfo.MobileNumber, entity.PhoneMobile);
            Assert.AreEqual(customer.ContactInfo.WorkNumber, entity.PhoneWork);
            Assert.AreEqual(customer.ContactInfo.Email, entity.Email);
            Assert.AreEqual(customer.ContactInfo.Address.Address1, entity.Address1);
            Assert.AreEqual(customer.ContactInfo.Address.Address2, entity.Address2);
            Assert.AreEqual(customer.ContactInfo.Address.City, entity.City);
            Assert.AreEqual(customer.ContactInfo.Address.State, entity.State);
            Assert.AreEqual(customer.ContactInfo.Address.Zip, entity.ZipCode);
            Assert.AreEqual(bankId, entity.BankId);
            Assert.AreEqual(branchId, entity.CustomerBranchId);
            Assert.AreEqual(typeId, entity.CustomerTypeId);
            Assert.AreEqual(statusId, entity.CustomerStatusId);
            Assert.AreEqual(riskId, entity.ClassificationId);
            Assert.AreEqual(officerId, entity.CustomerOfficerId);
            //Assert.AreEqual(time.Now, entity.ModifiedDate);
        }
    }
}
```

Best Content Width if your audience uses large displays


I am creating a new website based on WordPress with Elementor – content: photography

Images will be large, at least 1500 pixels high, and either 1010 pixels wide or 2280 pixels wide.

So the generally recommended website width of 1200 pixels or so is not enough.

I was thinking of a 2500 pixels content width.

Is this a mistake?

Could this be a problem for mobile? Elementor has a mobile adjustment setting where you can customize the main website how it will appear on mobile.

And: as designs are now liquid, does content width at all matter?

mesh – Which Terrain LOD algorithm should I use for super large terrain?

My game needs a terrain, the requirements are:

  1. Freely Zoom in & zoom out, like GoogleEarth. Max resolution when zooming in ~100 meter, Max range when zooming out ~2000km (a whole country scale).
  2. Freely fly over to any direction with any height; Freely rotate camera; Framerate should not a bottleneck for all those basic camera movement.
  3. Support large heightmap data. I got mine real world elevation data from NASA with 7.5arc resolution, around 30k * 15k for a whole country.
  4. Also I need to consider the spherical earth surface curvature other than using a planar map. But this should be easy if I just map each vertex into spherical coordinates. I am not build the whole planetary LOD so it’s not even a problem.

I noticed there are many different LOD algorithm out there. For example:

  • ROAM: very classic algorithm, most done on CPU;
  • Geomipmapping: store the whole heightmap with best resolution as vertex buffer, with different LOD in vertex index. Then determine LOD and draw range in vertex index at runtime.
  • CDLOD:
  • Geo Clipmapping: I am feeling excited about this one since it claims to load all 200k * 100k US heightmap into video card memory and compressed ~300M, it seems to meet all my requirements. see source paper. And this is paper in 2004 and think about the video card at that time!

That’s pretty much all I know about terrain LOD.
I am not familiar recently study/research in this area.
Is Geo clipmapping my best option? What else algorithms are also useful to consider?

performance – What’s the bottleneck in transfer of a large number of small files?

While taking backup of a collection of folder containing source code (where I realized later that I could exclude certain folders containing library files like node_modules), I noticed that the file transfer speed slows to a crawl (few kb/s against the usual 60 mb/s that the backup drive allows).

I’d like to understand where the bottleneck is. Is it some computation which must be performed which interleaves with the pure I/O and thus slows the whole thing down, or is it that the filesystem index on the target drive has some central lock which must be acquired and released between files?

I’m using NTFS on the target backup drive, and it’s a HDD.

sql server – Recommendations about deleting large set of rows in MSSQL

I need to delete about 75+ million rows from a table everyday that contains around 3.5 billions of record.

Database recovery mode is simple, I have writen a code that deletes 15.000 rows in a while condition until all 75M records is deleted. (i use batch delete due to log file grow) However, with current deletion speed it looks like it will take at least 5 days, which means that amount of data required to be deleted is multiply faster than my deletion speed.

Basically what i’m trying to do is summarizing (in another table) and deleting data older than 2 months. There is no update operation in that table, only insert and delete.

I have an enterprise edition of MSSQL 2017

Any suggestions will be welcome.

dnd 5e – Rules for DM’ing large scale battles

I am working on a campaign and there is a very good chance that the PC’s may find themselves involved in a large mass battle, a large siege or possibly both if things go slightly sideways in regards to there actions and success through the campaign (of course they may just as equally decide to get out of dodge if it all goes sideways but that is the joy of running a free from open campaign).

I have experience of a system for large battles from 1st edition legend of the 5 rings, a combination of the players attempting to complete heroic actions to sway the battle, combined with dice rolling the skill of the generals behind the scenes to gauge the flow of the battle. Are there any official published 5th edition rules or suggestions as to how to run these kind of large scale battles for D&D?

How to sort and search in javascript large set

The array “scores” tells the total points for each person involved in a contest. So for example:

User A: 100 points
User B: 90 points
User C: 90 points
User D: 80 points
User E: 75 points
User F: 60 points

According to above scores we also have this ranking:

User A: #1
User B: #2  
User C: #2
User D: #3
User E: #4
User F: #5

This ranking method follows the Dense Ranking method.

Then we have a user named alice. If she gets 55 points, she will rank at position #6 (according to ranking above).
If she scores 90 points, she will rank at position #2. And so on.

I actually have an array containing different “sessions” for alice. So having for example:
(55, 90)

This means that first time will alice be ranked at position #6. While second time she will be ranked at position #2.

I coded this, and it work. However, this does not seem to be very effective. For large datasets, with half million entries in the scores-array, it times out. This is the code:

const getPosition = (element, scores) => {
    scores.push(element);
    scores.sort(function (a,b) { return b-a; });
    return scores.indexOf(element)+1;
}

function climbingLeaderboard(scores, alice) {
    var uniqueSet = new Set(scores);
    scores = (...uniqueSet);
    var positions = ();
    let aliceIndex = 0;
    while(aliceIndex < alice.length){
        positions.push(getPosition(alice(aliceIndex), scores));
        aliceIndex++;
    }
    return positions;
}

function main() {
    const scores = (100, 90, 90, 80, 75, 60);
    const alice = (50, 65, 77, 90, 102);
    let result = climbingLeaderboard(scores, alice);
    console.log(result.join("n") + "n");
}

I guess the “sort”-function and/or searching for the element in the array with indexOf is the problem. But I could not find a way to make these two operations more efficient.