German Electricity Data :electric_plug:

• Data Source: SMARD Web Portal
• Task: forecasting (using the past to predict the future)
• Language: julia

First steps:

• load the data (with appropriate column types)
• split the data into train/val/test sets
• plot the time series (at different temporal resolution)
• calculate the mean production per month
• is there a seasonal pattern (what kinds of seasonalities)
• make a prediction for the validation set
• evaluate the quality of the prediction

using Random

Random.seed!(42)
"Jörg Marko Sascha Chaitanya Sonia Robin Andi" |> split |> shuffle |> x -> join(x," -> ")

"Jörg -> Sascha -> Robin -> Andi -> Chaitanya -> Marko -> Sonia"

1. Data loading

• load the hourly data from the file Realisierte_Erzeugung_201501010000_202305301300_Stunde.csv as a table into julia
• make sure, that data is interpreted and typed correctly
• clean header names

using CSV, DataFrames, Dates

function string_parse(strg)
    if ismissing(strg) 
        return strg
    else
    string = replace(strg, "." => "")
    newstring = replace(string, "," => ".")
    newnewstring = replace(newstring, "-" => missing)
    return newnewstring
    end
end

string_parse (generic function with 1 method)

function str_vec_to_float(vec) 
    b = string_parse.(vec)
    c = Vector{Union{Missing, Float64}}(undef,0)
    for (i,x) in enumerate(b) 
        if !ismissing(x) 
            push!(c, parse(Float64, x))
        else
            push!(c,x)
        end
    end
    return c
end

str_vec_to_float (generic function with 1 method)

df = DataFrame(CSV.File("Realisierte_Erzeugung_201501010000_202305301300_Stunde.csv", delim = ";"; missingstring = ["-"]))
head = names(df)
df[!,1] = Date.(df[!,1], "dd.mm.yyyy")
for i in head[4:end]
    df[!,i] = str_vec_to_float(df[!,i])
end

df

73716×15 DataFrame

73691 rows omitted

Row	Datum	Anfang	Ende	Biomasse [MWh] Berechnete Auflösungen	Wasserkraft [MWh] Berechnete Auflösungen	Wind Offshore [MWh] Berechnete Auflösungen	Wind Onshore [MWh] Berechnete Auflösungen	Photovoltaik [MWh] Berechnete Auflösungen	Sonstige Erneuerbare [MWh] Berechnete Auflösungen	Kernenergie [MWh] Berechnete Auflösungen	Braunkohle [MWh] Berechnete Auflösungen	Steinkohle [MWh] Berechnete Auflösungen	Erdgas [MWh] Berechnete Auflösungen	Pumpspeicher [MWh] Berechnete Auflösungen	Sonstige Konventionelle [MWh] Berechnete Auflösungen
	Date	Time	Time	Float64?	Float64?	Float64?	Float64?	Float64?	Float64?	Float64?	Float64?	Float64?	Float64?	Float64?	Float64?
1	2015-01-01	00:00:00	01:00:00	4024.25	1158.25	516.5	8128.0	0.0	133.0	10710.5	15687.2	3219.75	1226.25	1525.75	4909.25
2	2015-01-01	01:00:00	02:00:00	3982.75	1188.0	516.25	8297.5	0.0	122.5	11086.2	15321.8	2351.25	870.75	1079.25	4932.75
3	2015-01-01	02:00:00	03:00:00	4019.5	1139.25	514.0	8540.0	0.0	93.0	11026.2	14817.5	2227.0	809.5	787.0	5041.75
4	2015-01-01	03:00:00	04:00:00	4040.75	1122.5	517.75	8552.0	0.0	86.5	11027.8	14075.0	2339.75	821.0	287.75	5084.0
5	2015-01-01	04:00:00	05:00:00	4037.75	1112.0	519.75	8643.5	0.0	86.5	10962.2	14115.0	2461.5	831.25	346.75	5070.75
6	2015-01-01	05:00:00	06:00:00	4028.25	1107.75	520.0	8711.75	0.0	86.75	10696.0	13474.2	2217.75	851.0	765.5	5096.75
7	2015-01-01	06:00:00	07:00:00	4013.25	1111.75	521.5	9167.25	0.0	87.0	10299.5	12403.8	2373.25	868.25	414.5	5153.0
8	2015-01-01	07:00:00	08:00:00	4012.75	1113.75	520.25	9811.0	0.0	87.0	10035.2	12062.5	2491.0	876.0	582.5	5161.0
9	2015-01-01	08:00:00	09:00:00	3999.75	1107.5	525.25	9683.0	53.0	87.0	10245.8	12405.0	2530.25	888.25	750.5	5393.5
10	2015-01-01	09:00:00	10:00:00	4016.25	1121.0	527.0	9501.75	773.25	85.75	10060.2	12798.8	2386.25	891.5	387.0	5884.0
11	2015-01-01	10:00:00	11:00:00	4007.75	1122.0	525.0	10025.0	2116.75	82.0	10082.2	12728.5	2557.0	888.25	176.25	6064.0
12	2015-01-01	11:00:00	12:00:00	4011.75	1146.25	527.25	10862.5	3364.25	82.0	9949.0	12451.0	2657.75	876.5	884.5	5914.75
13	2015-01-01	12:00:00	13:00:00	4014.0	1139.75	527.75	11575.2	4198.25	82.0	10119.0	12464.5	2937.75	874.25	705.0	5778.25
⋮	⋮	⋮	⋮	⋮	⋮	⋮	⋮	⋮	⋮	⋮	⋮	⋮	⋮	⋮	⋮
73705	2023-05-30	01:00:00	02:00:00	4581.5	2033.75	3891.75	11488.5	0.0	119.5	0.0	3523.5	1288.5	3984.75	769.25	1146.0
73706	2023-05-30	02:00:00	03:00:00	4551.0	1963.5	3725.5	10141.0	0.0	119.0	0.0	3484.25	1263.75	3592.0	604.25	1141.75
73707	2023-05-30	03:00:00	04:00:00	4503.25	2090.5	3635.25	9170.25	0.0	119.0	0.0	3490.0	1272.5	4030.5	578.75	1132.5
73708	2023-05-30	04:00:00	05:00:00	4454.25	2105.25	3635.75	8735.75	2.25	120.0	0.0	3678.25	1275.0	4895.5	433.25	1136.0
73709	2023-05-30	05:00:00	06:00:00	4406.75	2134.75	2940.0	8419.75	407.75	120.25	0.0	4087.5	1289.25	5723.75	404.25	1142.25
73710	2023-05-30	06:00:00	07:00:00	4413.25	2227.5	2272.5	7531.0	2888.75	128.25	0.0	5010.75	1362.0	6525.25	2803.75	1144.5
73711	2023-05-30	07:00:00	08:00:00	4468.0	2276.5	1916.25	5507.25	8343.25	137.0	0.0	5275.5	1173.75	6905.75	5712.75	1156.75
73712	2023-05-30	08:00:00	09:00:00	4503.5	2152.5	1643.25	3647.75	15893.2	138.0	0.0	5181.75	1196.25	6946.5	5079.5	1132.0
73713	2023-05-30	09:00:00	10:00:00	4544.5	2156.25	1285.0	2900.5	23612.5	135.0	0.0	4707.5	1114.75	6384.5	2122.75	1115.0
73714	2023-05-30	10:00:00	11:00:00	4568.25	2085.25	1241.25	2729.5	30341.8	121.25	0.0	3978.5	1095.5	4802.5	1279.0	1096.25
73715	2023-05-30	11:00:00	12:00:00	4618.75	2068.25	1053.25	3115.0	35196.2	119.0	0.0	3759.75	1151.5	3762.25	774.0	1096.5
73716	2023-05-30	12:00:00	13:00:00	4612.25	1967.25	781.25	3496.25	37030.0	119.0	0.0	3765.0	1296.75	3240.0	196.75	1099.75