The other half of a data or ML interview in C#. It covers LINQ, streaming, CSV traps, hand-coded statistics, numeric stability, sampling, sketches, SIMD and ML.NET.
A data loop rarely stops at the patterns. It adds two questions. Can you shape data without writing a loop for every step? Can you code the maths without a library? This page covers both. Every compiled block here builds and is tested.
System.Numerics.Tensors and Microsoft.ML. Every method still names its cost, same as Guide 1.First decide which layer you work at. A wrong pick costs ten times the runtime. Interviewers watch for it.
for loop over an array or span.
IEnumerable<T>.
foreach runs the whole chain again.Vector<T> or TensorPrimitives.
List per row, and ReadAllLines on a 10 GB file are what sink a data solution. Stream it, keep values unboxed, and only then think about speed.IEnumerable<T> in System.Linq. Most are lazy: they build a pipeline and run nothing until you enumerate. ToList, Count, Sum and foreach force it to run.The data-shaping operators come in two groups. The split-apply-combine group is GroupBy, CountBy, AggregateBy and ToLookup. The reshape group is Chunk, Zip and Index. CountBy, AggregateBy and Index arrived in .NET 9.
/// <summary>One sales row. A record gives value equality and a readable ToString.</summary>
public sealed record Sale(string Region, string Product, int Units, decimal Price);
public static class Shaping
{
/// <summary>Total units per region, ordered by region name. O(n log g) for g groups.</summary>
/// <param name="sales">The rows to group.</param>
/// <returns>One (region, units) pair per region.</returns>
/// <example>UnitsByRegion([new("EU", "a", 2, 1m), new("EU", "b", 3, 1m)]): [(EU, 5)]</example>
public static List<(string Region, int Units)> UnitsByRegion(IEnumerable<Sale> sales) =>
sales.GroupBy(s => s.Region) // split: one bucket per region
.Select(g => (Region: g.Key, Units: g.Sum(s => s.Units))) // apply: reduce each
.OrderBy(p => p.Region, StringComparer.Ordinal) // sort so the output is stable
.ToList(); // combine: force it to run once
/// <summary>How often each word occurs, most frequent first. O(n + d log d).</summary>
/// <param name="words">The words to count.</param>
/// <returns>(word, count) pairs, ties broken by word.</returns>
/// <example>WordCounts(["b", "a", "b"]) gives [(b, 2), (a, 1)]</example>
public static List<(string Word, int Count)> WordCounts(IEnumerable<string> words) =>
words.CountBy(w => w) // .NET 9: no GroupBy buckets kept
.OrderByDescending(kv => kv.Value)
.ThenBy(kv => kv.Key, StringComparer.Ordinal) // tie-break so order is fixed
.Select(kv => (kv.Key, kv.Value))
.ToList();
/// <summary>Revenue per region in one pass, no buckets. O(n + g log g).</summary>
/// <param name="sales">The rows to fold.</param>
/// <returns>(region, revenue) pairs ordered by region.</returns>
/// <example>Revenue([new("EU", "a", 2, 1.5m)]) gives [(EU, 3.0)]</example>
public static List<(string Region, decimal Revenue)> Revenue(IEnumerable<Sale> sales) =>
sales.AggregateBy(
s => s.Region, // the group key
0m, // seed: revenue starts at zero
(total, s) => total + s.Units * s.Price) // fold one row into the total
.OrderBy(kv => kv.Key, StringComparer.Ordinal)
.Select(kv => (kv.Key, kv.Value))
.ToList();
/// <summary>Products sold in one region. Build O(n), then O(1) per lookup.</summary>
/// <param name="sales">The rows to index.</param>
/// <param name="region">The region to ask about. An unknown region is fine.</param>
/// <returns>The product names, in input order. Empty if the region is unknown.</returns>
/// <example>ProductsIn(sales, "Mars") gives [] rather than throwing</example>
public static List<string> ProductsIn(IEnumerable<Sale> sales, string region)
{
// ToLookup is eager: it reads every row now, unlike GroupBy.
ILookup<string, string> byRegion = sales.ToLookup(s => s.Region, s => s.Product);
// A missing key gives an empty sequence. A Dictionary would throw here.
return byRegion[region].ToList();
}
/// <summary>Sum of each batch of size items. O(n) time, O(size) memory.</summary>
/// <param name="values">The stream to batch.</param>
/// <param name="size">Batch size, at least 1. The last batch may be short.</param>
/// <returns>One sum per batch.</returns>
/// <example>BatchSums([1, 2, 3, 4, 5], 2) gives [3, 7, 5]</example>
public static List<int> BatchSums(IEnumerable<int> values, int size) =>
values.Chunk(size).Select(batch => batch.Sum()).ToList();
/// <summary>Change from each value to the next. O(n).</summary>
/// <param name="values">A series, for example daily totals.</param>
/// <returns>n - 1 deltas. Empty for 0 or 1 values.</returns>
/// <example>Deltas([3, 5, 4]) gives [2, -1]</example>
public static List<int> Deltas(IReadOnlyList<int> values) =>
// Zip stops at the shorter input, and Skip(1) is one shorter, so we get n - 1 pairs.
values.Zip(values.Skip(1), (previous, next) => next - previous).ToList();
/// <summary>Positions of the values above a threshold. O(n).</summary>
/// <param name="values">The values to scan.</param>
/// <param name="threshold">Values strictly above this are kept.</param>
/// <returns>The 0-based indexes that pass.</returns>
/// <example>IndexesAbove([5, 1, 9], 4) gives [0, 2]</example>
public static List<int> IndexesAbove(IEnumerable<int> values, int threshold) =>
values.Index() // .NET 9: (Index, Item) pairs
.Where(p => p.Item > threshold)
.Select(p => p.Index)
.ToList();
}
var q = rows.Where(...), then call q.Count() and q.ToList(). The filter runs twice. If rows reads a file, the file is read twice. Call ToList() once when you need the result more than once.GroupBy is lazy to start but not streaming. The first MoveNext reads the whole input into buckets. On a 10 GB stream it needs all 10 GB. CountBy and AggregateBy keep one value per key, so they need memory for the keys only. Use them when you only want a count or a total.AggregateBy here, not GroupBy then Sum. It keeps one running total per key, so memory is the number of keys, not the number of rows.”When data does not fit in memory, the answer is a pipeline of IEnumerable<T>. Each stage pulls one item at a time from the stage before it. File.ReadLines starts that pipeline. It yields one line at a time. File.ReadAllLines loads every line into an array first.
Reading the figure. Blue boxes are lazy stages. The green sink drives the work by asking for items. The red bar is the eager version. It must hold the whole file, and .NET strings use two bytes per character.
public static class Streaming
{
/// <summary>Lists of at most size items. O(size) memory, O(n) total.</summary>
/// <param name="items">The source. Read once, lazily.</param>
/// <param name="size">Batch size, at least 1.</param>
/// <returns>The batches. The last one may be short.</returns>
/// <example>Batches([0, 1, 2, 3, 4], 2) gives [[0, 1], [2, 3], [4]]</example>
public static IEnumerable<List<T>> Batches<T>(IEnumerable<T> items, int size)
{
// Check here, in a normal method, so a bad size throws at the call.
// Inside an iterator the throw would wait until the first MoveNext.
ArgumentOutOfRangeException.ThrowIfLessThan(size, 1); // 1: a batch must hold an item
return BatchesCore(items, size);
}
private static IEnumerable<List<T>> BatchesCore<T>(IEnumerable<T> items, int size)
{
var batch = new List<T>(size); // capacity size: never regrows
// Invariant: batch holds the items read since the last yield, fewer than size.
foreach (T item in items)
{
batch.Add(item);
if (batch.Count == size)
{
yield return batch;
batch = new List<T>(size); // a NEW list: callers may keep the old one
}
}
if (batch.Count > 0) // > 0: the final short batch, if any
yield return batch;
}
/// <summary>Drops repeats while streaming. O(distinct) memory.</summary>
/// <param name="items">The source, read once.</param>
/// <returns>Each value the first time it appears.</returns>
/// <example>RunningUnique([1, 2, 1, 3, 2]) gives [1, 2, 3]</example>
public static IEnumerable<T> RunningUnique<T>(IEnumerable<T> items)
{
var seen = new HashSet<T>();
foreach (T item in items)
{
if (seen.Add(item)) // Add returns false for a repeat
yield return item;
}
}
/// <summary>Sum of one numeric column of a CSV file, read lazily. O(lines).</summary>
/// <param name="path">The file. The first line is a header and is skipped.</param>
/// <param name="column">0-based column index.</param>
/// <returns>The total of that column.</returns>
/// <example>SumColumn("sales.csv", 2) gives 42.5 for rows a,b,40 and c,d,2.5</example>
public static double SumColumn(string path, int column) =>
File.ReadLines(path) // lazy: one line in memory at a time
.Skip(1) // 1: skip the header row
.Where(line => line.Length > 0) // a trailing newline gives an empty line
.Select(line => Csv.ParseNumber(Csv.ParseLine(line)[column]))
.Sum();
/// <summary>K-way merge of sorted runs, lazily. O(n log k), O(k) memory.</summary>
/// <param name="runs">Each run is sorted ascending.</param>
/// <returns>All values, sorted ascending.</returns>
/// <example>MergeSorted([[1, 4], [2, 3]]) gives [1, 2, 3, 4]</example>
public static IEnumerable<int> MergeSorted(IEnumerable<IEnumerable<int>> runs)
{
// The heap holds one live cursor per run, keyed by its current value.
var heap = new PriorityQueue<IEnumerator<int>, int>();
var open = new List<IEnumerator<int>>();
try
{
foreach (IEnumerable<int> run in runs)
{
IEnumerator<int> cursor = run.GetEnumerator();
open.Add(cursor); // remember it so finally can dispose it
if (cursor.MoveNext()) // an empty run never enters the heap
heap.Enqueue(cursor, cursor.Current);
}
// Invariant: the heap top is the smallest value not yet yielded.
while (heap.TryDequeue(out IEnumerator<int>? cursor, out int value))
{
yield return value;
if (cursor.MoveNext()) // refill from the same run
heap.Enqueue(cursor, cursor.Current);
}
}
finally
{
foreach (IEnumerator<int> cursor in open)
cursor.Dispose(); // file-backed runs close their handles
}
}
}
yield return runs none of its body at the call. A bad argument throws on the first MoveNext, which may be far away. Split it in two: a plain public method that validates, and a private iterator. Batches above does this. Chunk in the BCL does the same.Chunk is the built-in batcher. It yields T[] arrays and validates eagerly. Write your own in an interview to show you know the yield rules, then name Chunk. For async sources use IAsyncEnumerable<T> with File.ReadLinesAsync and await foreach.Array.Sort. Write it to a temp file. Cost O(n log m) for chunk size m.File.ReadLines. Feed them to MergeSorted. Cost O(n log k) for k runs.The same shape answers “sort 100 GB with 8 GB of RAM” and “merge k sorted files”. It is also the shuffle stage of map-reduce. Pattern 20 covers the heap in depth.
“Just split on commas” is the first bug. Name the traps before you write the parser.
"Smith, Jo",42 is two fields. Split(',') gives three."" means one literal quote character.double.Parse("1.5") uses the current culture. On a German machine that reads as 15. Always pass CultureInfo.InvariantCulture.File.ReadLines strips \r\n and a UTF-8 BOM. A raw Stream split by hand does not.using System.Globalization;
using System.Text;
public static class Csv
{
/// <summary>Splits one CSV line, honouring quotes. O(length).</summary>
/// <param name="line">One line, with no line terminator.</param>
/// <returns>The fields, quotes removed.</returns>
/// <example>ParseLine("\"Smith, Jo\",42") gives ["Smith, Jo", "42"]</example>
/// <exception cref="FormatException">A quote is opened and never closed.</exception>
public static List<string> ParseLine(string line)
{
var fields = new List<string>();
var field = new StringBuilder();
bool inQuotes = false;
// Invariant: field holds the current field so far, and inQuotes says
// whether we are inside a quoted section at index i.
for (int i = 0; i < line.Length; i++)
{
char c = line[i];
if (inQuotes)
{
if (c != '"')
field.Append(c); // commas are data inside quotes
else if (i + 1 < line.Length && line[i + 1] == '"')
{
field.Append('"'); // "" inside quotes is one quote
i++; // +1: skip the second quote
}
else
inQuotes = false; // closing quote
}
else if (c == '"')
inQuotes = true;
else if (c == ',')
{
fields.Add(field.ToString()); // a comma ends the field
field.Clear();
}
else
field.Append(c);
}
if (inQuotes)
throw new FormatException("unclosed quote");
fields.Add(field.ToString()); // the last field has no comma after it
return fields;
}
/// <summary>Parses a number the same way on every machine. O(length).</summary>
/// <param name="text">For example "1.5" or "-2e3".</param>
/// <returns>The value as a double.</returns>
/// <example>ParseNumber("1.5") gives 1.5, never 15</example>
public static double ParseNumber(string text) =>
double.Parse(text, NumberStyles.Float, CultureInfo.InvariantCulture);
/// <summary>Parses a "name,value" row without throwing. O(length).</summary>
/// <param name="line">The raw line.</param>
/// <param name="row">The parsed row when the result is true.</param>
/// <returns>False for a bad row, so the caller can count and skip it.</returns>
/// <example>TryParseRow("a,x", out _) gives false</example>
public static bool TryParseRow(string line, out (string Name, double Value) row)
{
row = default;
List<string> parts;
try { parts = ParseLine(line); }
catch (FormatException) { return false; }
if (parts.Count != 2) // 2: exactly name and value
return false;
if (!double.TryParse(parts[1], NumberStyles.Float,
CultureInfo.InvariantCulture, out double value))
return false;
row = (parts[0], value);
return true;
}
}
TryParse and count the rejects. One malformed line should not kill a ten-hour job. Report the reject count at the end, and fail only if it passes a threshold.Expect to write these without a library. Expect a follow-up on the numeric detail.
public static class Stats
{
/// <summary>Arithmetic mean. O(n) time, O(1) space.</summary>
/// <param name="values">At least one value.</param>
/// <returns>The sum divided by the count.</returns>
/// <example>Mean([1.0, 2.0, 6.0]) gives 3.0</example>
public static double Mean(IReadOnlyList<double> values)
{
if (values.Count == 0) // 0: no values, no mean
throw new ArgumentException("mean of an empty sequence");
double total = 0;
foreach (double v in values)
total += v;
return total / values.Count;
}
/// <summary>Middle value. The mean of the middle two for an even count.</summary>
/// <param name="values">At least one value. Not modified.</param>
/// <returns>The median. O(n log n) as written, O(n) expected with quickselect.</returns>
/// <example>Median([1.0, 2.0, 3.0, 4.0]) gives 2.5</example>
public static double Median(IReadOnlyList<double> values)
{
if (values.Count == 0)
throw new ArgumentException("median of an empty sequence");
double[] sorted = [.. values]; // copy: never sort the caller's data
Array.Sort(sorted);
int mid = sorted.Length / 2; // /2: the upper middle for even counts
return sorted.Length % 2 == 1 // %2 == 1: odd count has one middle
? sorted[mid]
: (sorted[mid - 1] + sorted[mid]) / 2; // -1: the lower middle; /2: average
}
/// <summary>Spread around the mean, two passes. O(n).</summary>
/// <param name="values">At least two values.</param>
/// <param name="sample">True divides by n - 1 (Bessel). False divides by n.</param>
/// <returns>The variance.</returns>
/// <example>Variance([2, 4, 4, 4, 5, 5, 7, 9], sample: false) gives 4.0</example>
public static double Variance(IReadOnlyList<double> values, bool sample = true)
{
int n = values.Count;
if (n < 2) // 2: one value has no spread to measure
throw new ArgumentException("variance needs at least two values");
double mu = Mean(values); // pass 1
double total = 0;
foreach (double v in values) // pass 2: squared distance from the mean
total += (v - mu) * (v - mu);
return total / (sample ? n - 1 : n); // n - 1: one degree of freedom used by mu
}
/// <summary>Square root of the variance, back in the data's units. O(n).</summary>
/// <param name="values">At least two values.</param>
/// <param name="sample">Same meaning as in Variance.</param>
/// <returns>The standard deviation.</returns>
/// <example>StdDev([2, 4, 4, 4, 5, 5, 7, 9], sample: false) gives 2.0</example>
public static double StdDev(IReadOnlyList<double> values, bool sample = true) =>
Math.Sqrt(Variance(values, sample));
/// <summary>The p-th percentile with linear interpolation. O(n log n).</summary>
/// <param name="values">At least one value.</param>
/// <param name="p">From 0 to 100. 50 is the median.</param>
/// <returns>The value at rank p / 100 * (n - 1), blended between neighbours.</returns>
/// <example>Percentile([10, 20, 30, 40], 50) gives 25.0</example>
public static double Percentile(IReadOnlyList<double> values, double p)
{
if (values.Count == 0)
throw new ArgumentException("percentile of an empty sequence");
ArgumentOutOfRangeException.ThrowIfLessThan(p, 0.0); // 0: the minimum
ArgumentOutOfRangeException.ThrowIfGreaterThan(p, 100.0); // 100: the maximum
double[] sorted = [.. values];
Array.Sort(sorted);
// Ranks run 0..n-1, so p = 100 lands on the last index.
double rank = p / 100.0 * (sorted.Length - 1);
int low = (int)Math.Floor(rank);
int high = (int)Math.Ceiling(rank);
double weight = rank - low; // how far rank sits past low, 0 to 1
return sorted[low] + (sorted[high] - sorted[low]) * weight;
}
/// <summary>Nearest-rank percentile: always a real data point. O(n log n).</summary>
/// <param name="values">At least one value.</param>
/// <param name="p">Above 0 and at most 100.</param>
/// <returns>The smallest value with at least p percent of data at or below it.</returns>
/// <example>NearestRank([15, 20, 35, 40, 50], 40) gives 20.0</example>
public static double NearestRank(IReadOnlyList<double> values, double p)
{
double[] sorted = [.. values];
Array.Sort(sorted);
// Ceiling of p% of n is a 1-based rank, so -1 turns it into an index.
int rank = (int)Math.Ceiling(p / 100.0 * sorted.Length);
return sorted[Math.Max(rank, 1) - 1]; // Max(.., 1): p near 0 still picks the min
}
}
PERCENTILE.INC default. Nearest rank always returns a real value, which latency dashboards like. Ask which one the interviewer wants, or state which one you chose.E[x²] - E[x]² subtracts two huge, nearly equal numbers. With values near 10⁹ the answer comes out wrong, and it can even be negative. That is catastrophic cancellation, a classic interview question.public sealed class RunningStats
{
private double _sumSquaredDeltas; // M2 in the literature
/// <summary>How many values have been added.</summary>
public long Count { get; private set; }
/// <summary>The running mean. 0 before the first value.</summary>
public double Mean { get; private set; }
/// <summary>Folds one value in. O(1) time and memory, so it works on endless streams.</summary>
/// <param name="value">The new observation.</param>
/// <example>Add 2, 4, 4, 4, 5, 5, 7, 9 then Mean gives 5.0</example>
public void Add(double value)
{
Count++;
double delta = value - Mean; // distance from the OLD mean
Mean += delta / Count;
// Old delta times the distance from the NEW mean. This mix keeps it stable.
_sumSquaredDeltas += delta * (value - Mean);
}
/// <summary>Variance dividing by n. 0 when empty.</summary>
/// <example>After 2, 4, 4, 4, 5, 5, 7, 9 it gives 4.0</example>
public double PopulationVariance => Count > 0 ? _sumSquaredDeltas / Count : 0.0;
/// <summary>Variance dividing by n - 1. 0 with fewer than 2 values.</summary>
/// <example>After 1, 2, 3, 4 it gives 1.6666666666666667</example>
public double SampleVariance => Count > 1 ? _sumSquaredDeltas / (Count - 1) : 0.0;
/// <summary>The naive formula, kept here only to show it failing.</summary>
/// <param name="values">The data.</param>
/// <returns>E[x squared] minus E[x] squared. Wrong for large, close values.</returns>
/// <example>NaiveVariance([1e9 + 4, 1e9 + 7, 1e9 + 13, 1e9 + 16]) is not 22.5</example>
public static double NaiveVariance(IReadOnlyList<double> values)
{
double sum = 0, sumSquares = 0;
foreach (double v in values)
{
sum += v;
sumSquares += v * v; // about 1e18: only ~16 digits survive
}
double mean = sum / values.Count;
return sumSquares / values.Count - mean * mean;
}
}
public static class Scaling
{
/// <summary>Rescales to the range 0 to 1. O(n).</summary>
/// <param name="values">The data.</param>
/// <returns>(v - min) / (max - min). All zeros for a constant input.</returns>
/// <example>MinMax([10.0, 20.0, 30.0]) gives [0.0, 0.5, 1.0]</example>
public static double[] MinMax(IReadOnlyList<double> values)
{
double low = values.Min(), high = values.Max();
if (high == low)
return new double[values.Count]; // all zeros: avoid dividing by zero
double span = high - low;
return values.Select(v => (v - low) / span).ToArray();
}
/// <summary>Rescales to mean 0 and standard deviation 1. O(n).</summary>
/// <param name="values">At least two values.</param>
/// <returns>(v - mean) / sigma. All zeros for a constant input.</returns>
/// <example>ZScore([2, 4, 4, 4, 5, 5, 7, 9]) is [-1.5, -0.5, -0.5, -0.5, 0, 0, 1, 2]</example>
public static double[] ZScore(IReadOnlyList<double> values)
{
double mu = Stats.Mean(values);
double sigma = Stats.StdDev(values, sample: false);
if (sigma == 0.0)
return new double[values.Count];
return values.Select(v => (v - mu) / sigma).ToArray();
}
}
double is 64-bit binary floating point, about 15 to 17 significant digits, and fast in hardware. float is 32-bit, about 7 digits. decimal is 128-bit base-10 with 28 to 29 digits. It is exact for values like 0.1 but runs in software, about 10 to 20 times slower.decimal. 0.1m + 0.2m == 0.3m is true. With double it is false. Or store cents in a long.double. decimal has no Math.Exp or Math.Log, and its range is only about 10²⁸.== on computed doubles. Use a relative and an absolute tolerance. double.Epsilon is the smallest positive double, about 5e-324. It is not a useful tolerance.double.NaN == double.NaN is false. Use double.IsNaN. A NaN in a sort key breaks the ordering.int wraps silently by default. Use long, or a checked block to throw OverflowException.1.0 / 0 is Infinity. 1 / 0 with ints throws DivideByZeroException. 1m / 0 throws too.public static class Numerics
{
/// <summary>Plain left-to-right sum. O(n). Error grows with n.</summary>
/// <param name="values">The terms.</param>
/// <returns>The rounded running total.</returns>
/// <example>NaiveSum of ten 0.1 values gives 0.9999999999999999</example>
public static double NaiveSum(IEnumerable<double> values)
{
double total = 0;
foreach (double v in values)
total += v;
return total;
}
/// <summary>Kahan compensated sum. O(n). Error stays near one rounding.</summary>
/// <param name="values">The terms.</param>
/// <returns>A much more accurate total.</returns>
/// <example>KahanSum of ten 0.1 values gives 1.0</example>
public static double KahanSum(IEnumerable<double> values)
{
double total = 0;
double lost = 0; // the low bits dropped so far
// Invariant: total + lost is the true sum of the values seen, to about 1 ulp.
foreach (double v in values)
{
double y = v - lost; // put back what was lost last time
double t = total + y; // big + small: low bits of y drop here
lost = (t - total) - y; // recover exactly what was dropped
total = t;
}
return total;
}
/// <summary>Evaluates a polynomial with Horner and fused multiply-add. O(degree).</summary>
/// <param name="coefficients">Highest power first. [2, 0, 1] means 2x^2 + 1.</param>
/// <param name="x">Where to evaluate.</param>
/// <returns>The value. Each step rounds once, not twice.</returns>
/// <example>Horner([2.0, 0.0, 1.0], 3.0) gives 19.0</example>
public static double Horner(IReadOnlyList<double> coefficients, double x)
{
double result = 0;
// Invariant: result is the polynomial of the first i coefficients at x.
foreach (double c in coefficients)
result = Math.FusedMultiplyAdd(result, x, c); // result * x + c, one rounding
return result;
}
/// <summary>log(sum(exp(v))) without overflow. O(n).</summary>
/// <param name="values">Log-space values. Empty gives negative infinity.</param>
/// <returns>peak + log(sum(exp(v - peak))). Every exponent is at most 0.</returns>
/// <example>LogSumExp([1000.0, 1000.0]) gives about 1000.693147</example>
public static double LogSumExp(IReadOnlyList<double> values)
{
if (values.Count == 0)
return double.NegativeInfinity; // log of an empty sum, log(0)
double peak = values.Max();
if (double.IsNegativeInfinity(peak))
return double.NegativeInfinity; // every term is exp(-inf) = 0
double total = 0;
foreach (double v in values)
total += Math.Exp(v - peak); // at most exp(0) = 1, so no overflow
return peak + Math.Log(total);
}
/// <summary>Scores to probabilities. O(n). Subtracting the max prevents overflow.</summary>
/// <param name="scores">At least one score.</param>
/// <returns>Non-negative values that sum to 1.</returns>
/// <example>Softmax([1.0, 2.0, 3.0]) gives about [0.09, 0.2447, 0.6652]</example>
public static double[] Softmax(IReadOnlyList<double> scores)
{
double peak = scores.Max(); // the factor cancels top and bottom
double[] exps = scores.Select(s => Math.Exp(s - peak)).ToArray();
double total = exps.Sum();
return exps.Select(e => e / total).ToArray();
}
/// <summary>1 / (1 + exp(-x)), branching so exp never overflows. O(1).</summary>
/// <param name="x">Any real value.</param>
/// <returns>A value from 0 to 1.</returns>
/// <example>Sigmoid(0.0) gives 0.5</example>
public static double Sigmoid(double x)
{
if (x >= 0) // 0: exp(-x) is at most 1 here
return 1.0 / (1.0 + Math.Exp(-x));
double e = Math.Exp(x); // x below 0: exp(x) is below 1
return e / (1.0 + e);
}
/// <summary>Tolerant equality for computed doubles. O(1).</summary>
/// <param name="a">First value.</param>
/// <param name="b">Second value.</param>
/// <param name="relative">Allowed error as a share of the larger size. 1e-9 by default.</param>
/// <param name="absolute">Floor for values near zero. 1e-12 by default.</param>
/// <returns>True when a and b are close.</returns>
/// <example>NearlyEqual(0.1 + 0.2, 0.3) gives true</example>
public static bool NearlyEqual(double a, double b,
double relative = 1e-9, double absolute = 1e-12) =>
Math.Abs(a - b) <= Math.Max(relative * Math.Max(Math.Abs(a), Math.Abs(b)), absolute);
/// <summary>Splits an amount into n parts that add back exactly. O(n).</summary>
/// <param name="amount">Money, for example 10.00m.</param>
/// <param name="parts">How many shares, at least 1.</param>
/// <returns>Shares in cents precision. The first ones get the leftover cents.</returns>
/// <example>SplitMoney(10.00m, 3) gives [3.34, 3.33, 3.33]</example>
public static decimal[] SplitMoney(decimal amount, int parts)
{
ArgumentOutOfRangeException.ThrowIfLessThan(parts, 1); // 1: cannot split into 0
long cents = (long)(amount * 100); // 100: cents per unit
long each = cents / parts;
long leftover = cents % parts; // % parts: cents that do not divide
var shares = new decimal[parts];
for (int i = 0; i < parts; i++) // i: share index
shares[i] = (each + (i < leftover ? 1 : 0)) / 100m; // 1: one extra cent each
return shares;
}
/// <summary>Adds two ints and throws on overflow instead of wrapping. O(1).</summary>
/// <param name="a">First term.</param>
/// <param name="b">Second term.</param>
/// <returns>a + b.</returns>
/// <example>CheckedAdd(int.MaxValue, 1) throws OverflowException</example>
public static int CheckedAdd(int a, int b) => checked(a + b);
}
Math.FusedMultiplyAdd buys. a * b + c rounds twice: once after the multiply and once after the add. FMA computes it exactly and rounds once. On modern CPUs it is one instruction. It helps in dot products, Horner, and the Kahan-like two-product tricks.inf or NaN after Math.Exp. Cause: exp of a large positive number. Fix: subtract the max first, or use log-sum-exp.E[x²] - E[x]². Fix: Welford, or two passes.int overflow wraps. Fix: long or checked.public static class Distance
{
/// <summary>Dot product. O(n).</summary>
/// <param name="a">First vector.</param>
/// <param name="b">Second vector, same length.</param>
/// <returns>The sum of a[i] * b[i].</returns>
/// <example>Dot([1.0, 2.0], [3.0, 4.0]) gives 11.0</example>
public static double Dot(ReadOnlySpan<double> a, ReadOnlySpan<double> b)
{
if (a.Length != b.Length)
throw new ArgumentException("vectors must have the same length");
double total = 0;
for (int i = 0; i < a.Length; i++) // i: the coordinate being multiplied
total = Math.FusedMultiplyAdd(a[i], b[i], total);
return total;
}
/// <summary>Straight-line (L2) distance. O(n).</summary>
/// <param name="a">First point.</param>
/// <param name="b">Second point, same length.</param>
/// <returns>The square root of the summed squared differences.</returns>
/// <example>Euclidean([0.0, 0.0], [3.0, 4.0]) gives 5.0</example>
public static double Euclidean(ReadOnlySpan<double> a, ReadOnlySpan<double> b) =>
Math.Sqrt(SquaredEuclidean(a, b));
/// <summary>Squared L2 distance. Ranks the same as Euclidean with no sqrt. O(n).</summary>
/// <param name="a">First point.</param>
/// <param name="b">Second point, same length.</param>
/// <returns>The summed squared differences.</returns>
/// <example>SquaredEuclidean([0.0, 0.0], [3.0, 4.0]) gives 25.0</example>
public static double SquaredEuclidean(ReadOnlySpan<double> a, ReadOnlySpan<double> b)
{
double total = 0;
for (int i = 0; i < a.Length; i++)
{
double d = a[i] - b[i];
total += d * d;
}
return total;
}
/// <summary>Cosine of the angle between two vectors, from -1 to 1. O(n).</summary>
/// <param name="a">First vector, not all zeros.</param>
/// <param name="b">Second vector, not all zeros.</param>
/// <returns>dot(a, b) / (|a| |b|). Ignores length, only direction counts.</returns>
/// <example>Cosine([1.0, 1.0], [2.0, 2.0]) gives 1.0</example>
public static double Cosine(ReadOnlySpan<double> a, ReadOnlySpan<double> b)
{
double normA = Math.Sqrt(Dot(a, a)), normB = Math.Sqrt(Dot(b, b));
if (normA == 0.0 || normB == 0.0)
throw new ArgumentException("cosine is undefined for a zero vector");
return Dot(a, b) / (normA * normB);
}
/// <summary>Set overlap: intersection size over union size. O(|a| + |b|).</summary>
/// <param name="a">First set.</param>
/// <param name="b">Second set.</param>
/// <returns>From 0 to 1. Two empty sets give 1.</returns>
/// <example>Jaccard([1, 2, 3], [2, 3, 4]) gives 0.5</example>
public static double Jaccard<T>(IReadOnlySet<T> a, IReadOnlySet<T> b)
{
if (a.Count == 0 && b.Count == 0)
return 1.0; // 1: two empty sets are identical
int common = a.Count(b.Contains);
return (double)common / (a.Count + b.Count - common); // union = |a| + |b| - overlap
}
}
Math.Sqrt is monotone. Squared distances give the same order and stay exact for integer inputs. K Closest Points uses this trick.Random. Random.Shared is thread-safe and needs no setup, but you cannot seed it. new Random(seed) gives a repeatable sequence on one machine, but is not thread-safe. Take a Random parameter so tests can pass a seeded one.public static class Sampling
{
/// <summary>k items chosen uniformly from a stream of unknown length.</summary>
/// <param name="stream">The source, read once. O(n) time, O(k) memory.</param>
/// <param name="k">Sample size, 0 or more.</param>
/// <param name="rng">The random source. Null means Random.Shared.</param>
/// <returns>Up to k items. All items if the stream is shorter than k.</returns>
/// <example>Reservoir([0, 1, 2], 5) gives [0, 1, 2]</example>
public static List<T> Reservoir<T>(IEnumerable<T> stream, int k, Random? rng = null)
{
ArgumentOutOfRangeException.ThrowIfNegative(k);
rng ??= Random.Shared;
var reservoir = new List<T>(k);
int index = 0; // index: 0-based position in the stream
// Invariant: reservoir is a uniform sample of the first index items.
foreach (T item in stream)
{
if (index < k)
reservoir.Add(item); // fill the first k slots
else
{
// Next's upper bound is EXCLUSIVE, so + 1 makes 0..index inclusive.
int slot = rng.Next(0, index + 1);
if (slot < k) // happens with chance k / (index + 1)
reservoir[slot] = item;
}
index++;
}
return reservoir;
}
/// <summary>Shuffles then cuts, repeatably for one seed. O(n).</summary>
/// <param name="rows">The data. Not modified.</param>
/// <param name="testFraction">Share for the test set, from 0 up to but not 1.</param>
/// <param name="seed">Fixed seed so two runs compare.</param>
/// <returns>The train rows and the test rows.</returns>
/// <example>Split(0..9, 0.3, 0) gives 7 train rows and 3 test rows</example>
public static (T[] Train, T[] Test) TrainTestSplit<T>(
IReadOnlyList<T> rows, double testFraction, int seed)
{
if (testFraction is < 0.0 or >= 1.0) // 1.0 would leave nothing to train on
throw new ArgumentOutOfRangeException(nameof(testFraction));
T[] shuffled = [.. rows];
new Random(seed).Shuffle(shuffled); // .NET 8: Fisher-Yates in place
int cut = (int)(shuffled.Length * (1.0 - testFraction)); // 1.0 - f: the train share
return (shuffled[..cut], shuffled[cut..]); // ranges on arrays make copies
}
}
public sealed class WeightedSampler
{
private readonly double[] _cumulative; // prefix sums of the weights
/// <summary>Builds the prefix sums. O(n).</summary>
/// <param name="weights">Non-negative, with a positive total.</param>
/// <example>new WeightedSampler([0.0, 1.0, 0.0]) only ever draws 1</example>
public WeightedSampler(IReadOnlyList<double> weights)
{
if (weights.Count == 0 || weights.Any(w => w < 0))
throw new ArgumentException("weights must be non-empty and non-negative");
_cumulative = new double[weights.Count];
double running = 0;
for (int i = 0; i < weights.Count; i++) // i: weight index
{
running += weights[i];
_cumulative[i] = running; // _cumulative[i] = sum of weights 0..i
}
if (running <= 0) // 0: nothing could ever be drawn
throw new ArgumentException("weights must sum to a positive number");
}
/// <summary>One weighted draw. O(log n).</summary>
/// <param name="rng">The random source.</param>
/// <returns>Index i with chance weights[i] / total.</returns>
/// <example>Draw(Random.Shared) gives 1 for weights [0, 1, 0]</example>
public int Draw(Random rng)
{
double target = rng.NextDouble() * _cumulative[^1]; // ^1: the last prefix is the total
// Upper bound: first index whose prefix is strictly above target.
int lo = 0, hi = _cumulative.Length - 1; // -1: last valid index
// Invariant: the answer is in lo..hi.
while (lo < hi)
{
int mid = lo + (hi - lo) / 2; // /2: midpoint without int overflow
if (_cumulative[mid] > target) hi = mid;
else lo = mid + 1; // +1: mid is too small, skip it
}
return lo;
}
}
Random.Next(min, max) excludes max. Next(0, index) in a reservoir is a silent bias: the newest item can never win slot index. Write index + 1 and say why.string.GetHashCode() in a sketch you save. .NET randomises string hashes per process. Two runs give different hashes, so a saved sketch is garbage on reload. Hash the bytes yourself. FNV-1a is short enough to write from memory.using System.Numerics;
public static class StableHash
{
/// <summary>64-bit FNV-1a over the chars, then a SplitMix64 finish. O(length).</summary>
/// <param name="text">The key.</param>
/// <param name="seed">Gives a different hash family per seed, for multiple rows.</param>
/// <returns>The same value on every run and every machine.</returns>
/// <example>Hash64("a", 0) == Hash64("a", 0) is always true</example>
public static ulong Hash64(string text, ulong seed = 0)
{
// 14695981039346656037 is the published FNV-1a 64-bit offset basis.
ulong h = 14695981039346656037UL ^ seed;
foreach (char c in text)
{
h ^= c;
h *= 1099511628211UL; // the published FNV 64-bit prime
}
// SplitMix64 finaliser: spreads FNV's weak low bits across all 64 bits.
h ^= h >> 30; // 30, 27, 31: the published shifts
h *= 0xBF58476D1CE4E5B9UL; // published multiplier 1
h ^= h >> 27;
h *= 0x94D049BB133111EBUL; // published multiplier 2
h ^= h >> 31;
return h;
}
}
public sealed class CountMinSketch(int width = 512, int depth = 4)
{
// width 512: error about 2 * total / width. depth 4: fails with chance about 1/2^4.
private readonly long[,] _table = new long[depth, width];
/// <summary>Adds count to key. O(depth), whatever the stream length.</summary>
/// <param name="key">The item.</param>
/// <param name="count">How much to add. 1 by default.</param>
/// <example>Add("clicks") five times then Estimate("clicks") gives 5</example>
public void Add(string key, long count = 1)
{
for (int row = 0; row < depth; row++) // row: one independent hash per row
_table[row, Column(key, row)] += count;
}
/// <summary>The smallest counter across rows. O(depth).</summary>
/// <param name="key">The item.</param>
/// <returns>Never below the true count. Sometimes above, from collisions.</returns>
/// <example>Estimate("never-seen") gives 0 on an empty sketch</example>
public long Estimate(string key)
{
long best = long.MaxValue; // MaxValue: any real counter is smaller
for (int row = 0; row < depth; row++)
best = Math.Min(best, _table[row, Column(key, row)]);
return best;
}
private int Column(string key, int row) =>
(int)(StableHash.Hash64(key, (ulong)row) % (ulong)width); // % width: a valid column
}
public sealed class HyperLogLog
{
private readonly int _precision; // p: the top p hash bits pick a register
private readonly byte[] _registers;
/// <summary>Creates 2^precision registers. Error is about 1.04 / sqrt(2^precision).</summary>
/// <param name="precision">From 4 to 16. 12 gives 4096 registers and about 1.6% error.</param>
/// <example>new HyperLogLog(12) uses 4 KB</example>
public HyperLogLog(int precision = 12)
{
// 4: fewer than 16 registers is too noisy. 16: 64 KB of registers is plenty.
ArgumentOutOfRangeException.ThrowIfLessThan(precision, 4);
ArgumentOutOfRangeException.ThrowIfGreaterThan(precision, 16);
_precision = precision;
_registers = new byte[1 << precision]; // 1 << p: 2^p registers
}
/// <summary>Folds one item in. O(1).</summary>
/// <param name="item">The item. Repeats change nothing.</param>
/// <example>Add("a") twice counts once</example>
public void Add(string item)
{
ulong h = StableHash.Hash64(item);
int index = (int)(h >> (64 - _precision)); // 64 - p: keep the top p bits
ulong rest = h << _precision; // the remaining 64 - p bits
// Rank = position of the first 1 bit. A long run of zeros means a rare hash.
int rank = Math.Min(BitOperations.LeadingZeroCount(rest) + 1, // +1: ranks start at 1
64 - _precision + 1); // cap: all zeros in the rest
if (rank > _registers[index])
_registers[index] = (byte)rank;
}
/// <summary>Estimates the number of distinct items. O(2^precision).</summary>
/// <returns>The estimate, with small-range correction.</returns>
/// <example>10000 distinct adds give an estimate within about 5%</example>
public double Estimate()
{
int m = _registers.Length;
double harmonic = 0;
int zeros = 0;
foreach (byte r in _registers)
{
harmonic += Math.Pow(2, -r); // 2^-rank: rare hashes add little
if (r == 0) zeros++;
}
// 0.7213 and 1.079 are the published bias constants for m of 128 or more.
double alpha = 0.7213 / (1 + 1.079 / m);
double raw = alpha * m * m / harmonic;
// 2.5 m: the published cut-off below which linear counting is more accurate.
if (raw <= 2.5 * m && zeros > 0)
return m * Math.Log((double)m / zeros);
return raw;
}
}
Reading the figure. Each row has its own hash, so a key lands in one cell per row. Blue cells hold only “clicks”. The red cell also holds a colliding key, so it reads too high. The green box takes the minimum, which drops the polluted row.
Max, so they shard and combine for free.Vector<T> in System.Numerics picks the widest size the CPU supports. Vector<double>.Count is 2 on 128-bit hardware such as ARM NEON, and 4 with AVX2.Reading the figure. Blue and violet are the two full vector blocks. Each takes one instruction. Amber cells do not fill a vector, so a plain loop handles them. Green is the accumulator: one partial sum per lane, added together at the end.
using System.Numerics;
public static class Simd
{
/// <summary>Sum with Vector of double lanes plus a scalar tail. O(n / lanes).</summary>
/// <param name="values">The numbers.</param>
/// <returns>The total. May differ from a plain loop in the last bits.</returns>
/// <example>Sum([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]) gives 55.0</example>
public static double Sum(ReadOnlySpan<double> values)
{
int width = Vector<double>.Count; // lanes per vector: 2, 4 or 8 by CPU
var acc = Vector<double>.Zero;
int i = 0;
// Invariant: acc holds lane-wise sums of values[0..i]. i is a multiple of width.
for (; i <= values.Length - width; i += width)
acc += new Vector<double>(values.Slice(i, width));
double total = Vector.Sum(acc); // fold the lanes into one number
for (; i < values.Length; i++) // tail: fewer than width items left
total += values[i];
return total;
}
/// <summary>Dot product with SIMD. O(n / lanes).</summary>
/// <param name="a">First vector.</param>
/// <param name="b">Second vector, same length.</param>
/// <returns>The sum of a[i] * b[i].</returns>
/// <example>Dot([1, 2, 3], [4, 5, 6]) gives 32.0</example>
public static double Dot(ReadOnlySpan<double> a, ReadOnlySpan<double> b)
{
if (a.Length != b.Length)
throw new ArgumentException("vectors must have the same length");
int width = Vector<double>.Count;
var acc = Vector<double>.Zero;
int i = 0;
for (; i <= a.Length - width; i += width)
acc += new Vector<double>(a.Slice(i, width)) * new Vector<double>(b.Slice(i, width));
double total = Vector.Sum(acc);
for (; i < a.Length; i++)
total += a[i] * b[i];
return total;
}
/// <summary>Counts values above a threshold, 8 ints per step on AVX2. O(n / lanes).</summary>
/// <param name="values">The numbers.</param>
/// <param name="threshold">Strictly greater values count.</param>
/// <returns>How many values pass.</returns>
/// <example>CountAbove([1, 5, 9, 2, 8], 4) gives 3</example>
public static int CountAbove(ReadOnlySpan<int> values, int threshold)
{
int width = Vector<int>.Count;
var limit = new Vector<int>(threshold); // the threshold copied into every lane
var counts = Vector<int>.Zero;
int i = 0;
for (; i <= values.Length - width; i += width)
{
// GreaterThan gives -1 (all bits set) per passing lane, 0 otherwise.
counts -= Vector.GreaterThan(new Vector<int>(values.Slice(i, width)), limit);
}
int total = Vector.Sum(counts);
for (; i < values.Length; i++)
if (values[i] > threshold) total++;
return total;
}
}
Vector.IsHardwareAccelerated. If it is false, Vector<T> still works but runs in software. For fixed widths use Vector128<T>, Vector256<T> and Vector512<T> in System.Runtime.Intrinsics. The JIT also auto-vectorises some BCL calls: span.IndexOf, Sum on int[], and SequenceEqual are already SIMD.TensorPrimitives gives you the vectorised loops ready-made: sum, dot, cosine, softmax, and more over spans. It lives in the System.Numerics.Tensors NuGet package, not the shared framework. That is why this block is marked as not compiled.
// Needs: dotnet add package System.Numerics.Tensors
using System.Numerics.Tensors;
public static class TensorDemo
{
public static (float Sum, float Cosine, float[] Probabilities) Run()
{
float[] a = [1f, 2f, 3f, 4f];
float[] b = [4f, 3f, 2f, 1f];
float sum = TensorPrimitives.Sum(a); // SIMD sum, 10
float cosine = TensorPrimitives.CosineSimilarity(a, b); // 20 / 30 = 0.667
float[] probabilities = new float[a.Length]; // same length as the input
TensorPrimitives.SoftMax(a, probabilities); // stable softmax into dest
return (sum, cosine, probabilities);
}
}
TensorPrimitives.Dot. It picks AVX-512, AVX2 or NEON at run time and handles the tail for me.”public static class Classic
{
/// <summary>k-nearest-neighbours vote. O(n d) to score, O(n log n) to rank.</summary>
/// <param name="train">Training points, each of length d.</param>
/// <param name="labels">One label per training point.</param>
/// <param name="query">The point to classify.</param>
/// <param name="k">Neighbours to vote, from 1 to n.</param>
/// <returns>The most common label among the k nearest. Ties go to the nearer one.</returns>
/// <example>Knn([[0], [1], [10], [11]], ["lo", "lo", "hi", "hi"], [9.5], 3) is "hi"</example>
public static string Knn(double[][] train, string[] labels, double[] query, int k)
{
if (k < 1 || k > train.Length) // 1..n: need at least one voter
throw new ArgumentOutOfRangeException(nameof(k));
// Squared distance ranks the same and skips the sqrt.
// For large n, a size-k max-heap is O(n log k) and beats this full sort.
return Enumerable.Range(0, train.Length)
.OrderBy(i => Distance.SquaredEuclidean(train[i], query))
.Take(k)
.Select((i, rank) => (Label: labels[i], Rank: rank))
.GroupBy(v => v.Label)
.OrderByDescending(g => g.Count())
.ThenBy(g => g.Min(v => v.Rank)) // tie: the label with the nearest point
.First().Key;
}
/// <summary>Least-squares line y = w x + b, exact, one pass after the means. O(n).</summary>
/// <param name="xs">Inputs, not all equal.</param>
/// <param name="ys">Targets, same length.</param>
/// <returns>Slope w and intercept b.</returns>
/// <example>FitLine([0, 1, 2, 3], [1, 3, 5, 7]) gives (2.0, 1.0)</example>
public static (double W, double B) FitLine(IReadOnlyList<double> xs, IReadOnlyList<double> ys)
{
if (xs.Count != ys.Count || xs.Count == 0)
throw new ArgumentException("xs and ys must be non-empty and the same length");
double meanX = Stats.Mean(xs), meanY = Stats.Mean(ys);
double covariance = 0, spread = 0;
for (int i = 0; i < xs.Count; i++) // i: one (x, y) pair
{
covariance += (xs[i] - meanX) * (ys[i] - meanY);
spread += (xs[i] - meanX) * (xs[i] - meanX);
}
if (spread == 0.0)
throw new ArgumentException("all x values are equal, the slope is undefined");
double w = covariance / spread; // w = cov(x, y) / var(x)
return (w, meanY - w * meanX); // the line passes through the means
}
/// <summary>Precision, recall and F1 for 0/1 labels. O(n).</summary>
/// <param name="actual">True labels, 0 or 1.</param>
/// <param name="predicted">Model labels, 0 or 1, same length.</param>
/// <returns>Each value from 0 to 1. A zero denominator gives 0.</returns>
/// <example>PrecisionRecallF1([1, 1, 0, 0], [1, 0, 1, 0]) gives (0.5, 0.5, 0.5)</example>
public static (double Precision, double Recall, double F1) PrecisionRecallF1(
IReadOnlyList<int> actual, IReadOnlyList<int> predicted)
{
if (actual.Count != predicted.Count)
throw new ArgumentException("label lists must be the same length");
int tp = 0, fp = 0, fn = 0;
for (int i = 0; i < actual.Count; i++)
{
if (predicted[i] == 1 && actual[i] == 1) tp++; // 1: the positive class
else if (predicted[i] == 1) fp++;
else if (actual[i] == 1) fn++;
}
double precision = tp + fp > 0 ? (double)tp / (tp + fp) : 0.0;
double recall = tp + fn > 0 ? (double)tp / (tp + fn) : 0.0;
double f1 = precision + recall > 0
? 2 * precision * recall / (precision + recall) // 2: harmonic mean of two values
: 0.0;
return (precision, recall, f1);
}
}
tp / (tp + fp) with two ints is 0 for any value below 1. Cast one side to double first. This bug survives code review more often than you would think.ML.NET is Microsoft’s ML library for .NET. It trains and serves classic models in-process: regression, classification, ranking, anomaly detection. It also runs ONNX models exported from PyTorch. It is a NuGet package, Microsoft.ML, so the block below is marked as not compiled.
// Needs: dotnet add package Microsoft.ML
using Microsoft.ML;
using Microsoft.ML.Data;
public sealed class House
{
public float Size { get; set; }
public float Rooms { get; set; }
[ColumnName("Label")] public float Price { get; set; }
}
public sealed class PricePrediction
{
[ColumnName("Score")] public float Price { get; set; }
}
public static class MlNetDemo
{
public static float TrainAndPredict(IEnumerable<House> rows)
{
var ml = new MLContext(seed: 0); // seed 0: repeatable training
IDataView data = ml.Data.LoadFromEnumerable(rows);
var split = ml.Data.TrainTestSplit(data, testFraction: 0.2); // split BEFORE fitting
var pipeline = ml.Transforms
.Concatenate("Features", nameof(House.Size), nameof(House.Rooms))
.Append(ml.Transforms.NormalizeMinMax("Features")) // scaler fitted on train only
.Append(ml.Regression.Trainers.Sdca());
ITransformer model = pipeline.Fit(split.TrainSet);
var metrics = ml.Regression.Evaluate(model.Transform(split.TestSet));
Console.WriteLine($"RMSE {metrics.RootMeanSquaredError:F2}");
var engine = ml.Model.CreatePredictionEngine<House, PricePrediction>(model);
return engine.Predict(new House { Size = 120, Rooms = 3 }).Price;
}
}
PredictionEngine is not thread-safe. In ASP.NET use PredictionEnginePool from Microsoft.Extensions.ML. This is the follow-up question if you mention serving.GroupBy: O(n) time, O(n) memory. CountBy and AggregateBy: O(n) time, O(keys) memory.OrderBy: O(n log n), and it buffers the whole input. It is a stable sort. Array.Sort is not.ToLookup or ToDictionary: O(n) to build, O(1) per lookup.File.ReadLines and lazy LINQ.ToList.CountBy and AggregateBy over GroupBy when you only need a number per key.yield body.CultureInfo.InvariantCulture, and never split CSV on a bare comma.Math.Exp, decimal for money.Random.Next(min, max) excludes max. Take a Random parameter so tests can seed it.string.GetHashCode(). It changes every process. Hash the bytes yourself.