Split a list of points into a list of lines in C#

I have a list of points (System.Drawing.Point) obtained from an edge finder function that scanned a bitmap for objects. Each point is guaranteed to be a part of an edge line. I need to split the list into a collection of lines so additional math can be performed on them.

This is how I originally split the list:

public static List<List<Point>> Split(this List<Point> EdgesCoordinateList)
{
    if (EdgesCoordinateList is null) { throw new ArgumentNullException(nameof(EdgesCoordinateList), NullParameterErrorMessage); }
    if (EdgesCoordinateList.Count <= 1) { return null; }

    //  Link the line coordinates together into lines.
    //  Start by creating a new line with the first coordinate to act as a seed.
    List<List<Point>> Results = new List<List<Point>>
    {
        new List<Point> { EdgesCoordinateList(0) }
    };
    EdgesCoordinateList.RemoveAt(0);

    //  Iterate through the coordinates and assign each one to a line.
    foreach (Point TestPoint in EdgesCoordinateList)
    {
        //  Look through each previously found lines and see if the coordinate under test belongs to one of them.
        //  If coordinate under test is adjacent to one of the coordinates in the line under test, it belongs to that line.
        bool FoundLine = false;
        foreach (List<Point> LineList in Results.Where(LineList
            => LineList.FindLastIndex(i => Math.Abs(TestPoint.X - i.X) <= 1 && Math.Abs(TestPoint.Y - i.Y) <= 1) > -1))
        {
            //  Add the coordinate to the line it belongs to, then break the line search.
            Results(Results.IndexOf(LineList)).Add(TestPoint);
            FoundLine = true;
            break;
        }

        //  If line is not a part of preexisting lines then its the start of a new one.
        if (!FoundLine)
        { Results.Add(new List<Point> { TestPoint }); }
    }

    return Results;
}

It was great because it was fairly quick and worked well, except it messed up on lines that were relatively horizontal.

This is my second attempt to try to fix the issue:

public static List<List<Point>> Split(this List<Point> EdgesCoordinateList)
{
    if (EdgesCoordinateList is null) { throw new ArgumentNullException(nameof(EdgesCoordinateList), NullParameterErrorMessage); }
    if (EdgesCoordinateList.Count <= 1) { return null; }

    //  Link the line coordinates together into lines.
    //  Start by creating a new line with the first coordinate to act as a seed.
    List<List<Point>> Results = new List<List<Point>>
    {
        new List<Point> { EdgesCoordinateList(0) }
    };
    EdgesCoordinateList.RemoveAt(0);

    do
    {
        //  Copy the currently found lines into a list for iterating.
        //  Outer list is a list of lines.  Inner list is a list of the points in each line.
        List<List<Point>> FoundLines = Results.ConvertAll(item => new List<Point>(item));
        bool NoMoreMatches = true;

        //  Iterate through the list of lines.
        for (int i = 0; i < FoundLines.Count; i++)
        {
            //  Iterate through each point in the current line.
            foreach (Point TestPoint in FoundLines(i))
            {
                //  Find all of the points in the master list that are adjacent to the current test point.
                List<Point> FoundCoords = EdgesCoordinateList.FindAll(i => Math.Abs(TestPoint.X - i.X) <= 1 && (Math.Abs(TestPoint.Y - i.Y) <= 1));

                if (FoundCoords.Count > 0)
                {
                    //  Add all of the adjacent points to the current line.
                    Results(i).AddRange(FoundCoords);

                    //  Then remove them from the master list.
                    foreach (Point Coord in FoundCoords)
                    { _ = EdgesCoordinateList.Remove(Coord); }

                    NoMoreMatches = false;
                }
            }
        }

        //  When there are no more adjacent points to any of the points in the current line, make another line.
        if (NoMoreMatches)
        {
            Results.Add(new List<Point> { EdgesCoordinateList(0) });
            EdgesCoordinateList.RemoveAt(0);
        }

        //  Continue until there are no more points left in the master list.
    } while (EdgesCoordinateList.Count > 0);

    return Results;
}

The new solution solved the horizontal issue, but runs way too slow. (It went from less than a second to several minutes.)

How can I refactor this to speed it up? I would prefer it to work at least at a similar speed as the first method if possible.