blctools 0.0.44

Tools to work with data inside BLCs MS Sharepoint Cloud & Database

BLC Data Analytics tools package

Data Analytics helper functions to work with inside BLC's Cloud system.

Changelog

Version 0.0.43

• DatosCROM()
  • Improved: elaborar_curvas_de_potencia can now accept other dataframes.

Version 0.0.43

• DatosSMEC()
  • Improved: estimar_ENS_xxxx now runs orders between 10x (metodo='cuantil') and 40x (metodo='promedio') faster. Now they both run equally fast.

Version 0.0.42

• DatosSMEC()
  • Fixed bug: Initialization was not working properly.

Version 0.0.41

• DatosCROM()
  • Improved: all functions to estimate ENS were improved drastically. They're now 4x faster.

Version 0.0.40

• DatosCROM()
  • Fixed bug: buscar_indisponibilidades() was not working properly
  • Fixed bug: buscar_limitaciones() was not working properly
  • Fixed bug: buscar_incidencias() was not working properly

Version 0.0.39

• DatosCROM()
  • Fixed bug: consolidar_todo() was not categorizing internal/external incidents correctly for some reason

Version 0.0.38

• DatosCROM()
  • Fixed bug: exportar() really slow
  • Improved: explotar_incidencias() takes only incidencts with different start and end dates as input
  • Improved: Some functions were being executed twice upon object initialization

Version 0.0.37

• fechas
  • Fixed bug: mes_ant_dia_1() wasn't working properly.

Version 0.0.36

• DatosCROM()

  • Fixed bug: incidencias_resumen_status wasn't working properly

• TablasVC()

  • Fixed bug: Functions .consultar_.... where not working with "uc" parameter

Version 0.0.35

• DatosCROM()

  • Added: incidencias_resumen_status attribute
  • Deleted: Client attribute
  • Changed: resumen_diario_incidencias to incidencias_resumen_diario

• TablasVC()

  • Deleted: Client attribute

• PPO()

  • Changed: Default dates on initialization are now blctools.ayer()

• DTE()

  • Changed: Default dates on initialization are now blctools.ayer()
  • Changed: Improved internal modilarity
  • Changed: "dolarizar" parameter to "cols_usd"
  • Added: "dolarizar" method

Version 0.0.34

• BCRA()
  • Fixed bug: rem 24 month projections were not being projected properly

Version 0.0.33

• BCRA()
  • Added: cargar_tc_min = False as initialization parameter
  • Added: tc_min as the retail price for the USD in ARS
  • Fixed bug: rem projections were not being projected properly

Version 0.0.32

• fechas
  • Fixed bug: restar_mes wasn't working for some months of the year.

Version 0.0.31

• ArchivoMDB()

  • New object: It serves as an easy way to explore .mdb files

• ReporteBase()

  • Changed: how the object processes mdb files. It's now more efficient

Version 0.0.30

• BCRA()

  • Changed: On initializing, rem tryies to download all files
  • Changed: nor .rem_tc table is called .tc_rem
  • Changed: now obtener_rem_mensual() is called cargar_rem()
  • All possible attributes are nor .tc_dia, .tc_udh, .tc_rem and .rem

• DTE()

  • Changed: Cargar DTE goes on even if tc_udh couldn't be loaded from BCRA() object

• DatosSMEC()

  • Changed: consultar_datos_vc() is now cargar_datos_vc()

Version 0.0.29

• DTE()

  • Added: Option to convet prices to USD
  • Added: Initialization parameters cargar (bool), descargar (bool) and dolarizar (str,list,tuple,set)

• .fechas

  • Added: lista_mensual. Returns a list of dates provided a starting date and an "n"
  • Added: convertir_año_a_fecha() Returns a dt.DateTime object, starting from an integer (year)
  • Fixed bug: mes_ult_dia wasn't allowing for future dates

• BCRA()

  • Added: Added interface with Argentina's Central Bank to download exchange rate (tc) and Market Expectations (REM)

• DatosCROM()

  • Fixed bug: After 0.0.25 update, the object wasn't reading .xlsx files properly, thus crashing when trying to read.

Version 0.0.28

• DatosSMEC()
  • Changed: FechaOp column is now datetime type, when Querying VisionCROM's database.

Version 0.0.27

• DatosSMEC()
  • Fixed bug: HoraOp on cargar_datos_prn was not being calculated correctly.
  • Fixed bug: Actaris meters have now the right column order.
  • Changed: Now E_Neta_Gen and E_Neta_Con are calculated when each file is read, for consistency.

Version 0.0.26

• DatosCROM()

  • Added .DatosCROM() now warns if there is an empty excel file

• DatosCdO()

  • Added strings report
  • Changed the variable-reading process, made it more efficient. By adding .pickle file as backup.
  • Added listado_señales_strings(): get list of "string" signals (actually DC combiner boxes)
  • Added descargar_datos_strings(): download "string" signals' mesaurements (actually DC combiner boxes)

• DatosSMEC()

  • Added this new object to interact with VisionCROM's DataBase and fetch SMEC measurements
    • Initialization parameters:
      • fecha_i = Data's start date. DateTime object with year, month, day, hour and minute interval.
      • fecha_f = Data's end date. Same as fecha_i
      • parques = CAMMESAS's Nemotécnicos to indicate from which park the measurements will be retrieved
      • dir_salida = export directory
      • cargar_datos_basicos = sames as TablasVC()
      • mensajes = Verbose while loading
      • mensajes_SQL = Verbose while connecting
  • Added consultar_datos_vc(): Query VisionCROM's database using fecha_i, fecha_f and parque attributes
  • Added leer_archivo_prn(file): Read a single .prn file
  • Added cargar_datos_prn(folder): Read every .prn file in a given folder

Version 0.0.25

• TablasVC()

  • Changed: modificar_incidencia() now doesn't affect .incidencias_todas_auditoria attribute.

• DatosCROM()

  • Improved: .estimar_ENS_varias_incidencias()'s overall speed and precision. Still not good for production, but better for single WTG incidents.
  • Improved: Memory Usage overall
  • Removed : Pgen_U variable, was a duplicate of FC.
  • Removed: Pcon_U, Egen_U, Econ_U, Epos_U, very rarely useed, could be calculated by hand.

• DatosCdO()

  • Added this new object to interact with InAccess' API and download raw data from Caldenes del Oeste solar PV Plant (CdO)
    • Initialization parameters:
      • usr = Username (SCADA's username)
      • pwd = PAssword (SCADA's password)
      • apikey = Private apikey provided from InAccess
      • fecha_i = Data's start date. DateTime object with year, month, day, hour and 10 minute interval.
      • fecha_f = Data's end date. Same as fecha_i
      • reporte_produccion = True / False. Indicate if production report should be downloaded automatically
      • reporte_trackers = True / False. Indicate if trackers report should be downloaded automatically
      • reporte_mensual = True / False. If True, downloads both reports. False has no effect.
      • exportar = True / 'todos' / 'produccion / 'trackers'.
        • If True or 'todos' exports everything.
        • 'produccion' and 'trackers' export the respective report each.
  • Added obtener_señales(): Takes in a dataframe of Measurement Locations (Mlocs) and fetches all possible signals.
  • Added obtener_mediciones(): Takes a dataframe of signal lists and fetches data
  • Added listado_señales_trackers(): Just a preset, that shows which signals are used for the monthly production report
  • Added listado_señales_produccion(): Just a preset, that shows which signals are used for the monthly tracker report
  • Added mediciones_disponibles(): Initializing function that loads every available signal, from BLC's cloud
  • Added descargar_datos_produccion(): Downloads data from listado_señales_produccion() signals using DatosCdO's start and end date parameters
  • Added descargar_datos_trackers(): Downloads data from listado_señales_trackers() signals using DatosCdO's start and end date parameters
  • Added exportar(reportes = str): Exports loaded data from preset reports ['todos', 'produccion', 'trackers'].

Version 0.0.24

• DatosCROM()
  • estimar_ENS() now subtracts actual generated power from estimated power loss
  • Fixed bug: estimar_ENS() wasn't checking datatypes properly

Version 0.0.23

• PPO() and DTE()

  • Fixed bug: Setting "filtro" on initialization had no effect on running functions with default parameters.

• cl_CarpetaServicios()

  • Added: blctools now finds "Servicios" folder, even when it's not explicitly written in the OS Path.

• DatosCROM()

  • Added: estimar_ENS() takes a list of ids and estimates the ENS for each incident, based on actual production values.
  • Added: estimar_ENS_incidencias_abiertas() fills the ENS of the incidents that are still open
  • Added: estimar_ens_inc_abiertas = False initialization parameter
  • Added: incidencias_a_editar = None, takes a dictionary of dictionaries. The first dict has the IDs of the incidents that have to be edited as keys and their corresponding values are a second dictionary that holds the new parameters (and new values) . Example: {18088:{ENS=800,Owner='NewOwner'},18089:{Hours=8,Origin='INT'}}
  • Fixed bug: Plant Active Power and Energy was being calculated from WTGs / Circuits and it shouldn't. It has it's own values
  • Excludes incidents with Hours or ENS < 0 on .incidencias and .incidencias_redux attributes, but not in incidencias_todas
  • cargar_incidencias is now False as default.
  • explotar_incidencias is now False as default.

• Fechas module

  • Fixed bug: hora_op, fecha_op, sumar_mes / restar_mes now have prevenir_futuro=False as default parameter and pass it to validar_fecha(s)
  • Fixed bug: obtener_periodo now has prevenir_futuro=True as default parameter and passes it to mes_periodo(s)

• TablasVC()

  • Added .consultar_agrupamiento_de_un_equipo()
  • Added .modificar_incidencia() to edit a current incident
  • Fixed Bug: Query for incidencias was not returning open incidents

Version 0.0.22

• DatosCROM()
  • Added: One can now initialize the "parques" parameter with a single string, to avoid using brackets when working with just one park.
  • Fixed bug: Eerror when no active incidents were found for a given parq/period and trying to .interpretar_incidencias_bajo_iec61400()
  • Fixed bug: Some functions where trying to call TablasVC().consultar_incidencias() which doesn't exist anymore. Now its .consultar(incidencias='offline')

Version 0.0.21

• DatosCROM()

  • Added: .buscar_indisponibilidades(). Searches for periods of time which the WTGs met certain condition filters of out of service
  • Added: .buscar_limitaciones(). Searches for periods of time which the PLANT or WTGs had an SP_P > than the max.
  • Added: .buscar_incidencias() combines both previous functions in a single line.
  • Added: .incidencias_autodetectadas attribute, with export capabilities
  • Added: "Status" column to incidencias_redux
  • Added: "Count" and "Histogram" columns when calculating powercurves. _count "sheet" on export is now always there.
  • Added: "explotar_incidencias" initialization parámeter, which can take either 'iec61400' (default) or 'crudas'
  • Fixed bug: Missing Indices on datos_seg when trying to consolidate incidents and 10second data
  • _identificar_indisponibilidades() pgen and ppos parameters were added, to further improve filtering capacity
  • __procesar_d10s_dc() and cargar_segundales() now use DataFrame.reindex() to fill with blank rows where necessary.
  • elaborar_curvas_de_potencia() now sorts index clockwise instead of alphabetically

• TablasVC()

  • Added .consultar_agrupamiento_de_un_equipo()

Version 0.0.20

• DatosCROM()
  • interpretar_incidencias_bajo_iec61400() made more efficient, while keeping compatibility with pandas >= 1.5.0
  • _identificar_indisponibilidades() pgen and ppos parameters were added, to further improve filtering capacity
• TablasVC()
  • Added "potencia=False" parameter to consultar_agrupamientos(), consultar_equipos_no_agrupamientos()

Version 0.0.19

• DatosCROM()
  • Fixed bug: interpretar_incidencias_bajo_iec61400() was experiencing problems with pandas >= 1.5.0, and had a problem with the index selection statement at the beginning.
  • Added experimentally: _identificar_indisponibilidades(duracion=30)

Version 0.0.18

• DatosCROM()
  • .exportar() now flattens MultiIndex columns, freezes panes and formats floats, for easier view on Excel.
  • Added "invalidar_datos_seg_congelados" initialization parameter (default = False)
  • Fixed bug: .exportar() throwing error when curvas_de_potencia or consolidado held more than 1 Dataframe
  • Fixed bug: .interpretar_incidencias_bajo_iec61400() now trims final result between fecha_i and fecha_f
  • Fixed bug: .elaborar_curvas_de_potencia() wasn't working properly with filtro_n >1

Version 0.0.17

• fechas submodule
  • Added .sem_dia_1(date,offset=0) function. Returns first day of week of the given date + given offset in days (offset can be negative)
  • Added .sem_act_dia_1(offset=0) function. Returns first day of current week + given offset in days
  • Added .año_op(fecha_actual,fecha_cod) function. Returns the current contractual year based on the two dates (First year is 1, not 0)
• DatosCROM()
  • adjusted how xlsx files processing time is calculated
  • adjusted how .pickle files are created/read. Compression was added through bz2
  • consolidado attribute is now called consolidado_redux, which is a subset of columns of consolidado
  • consolidado attribute now holds all the original data, but resampled to the users' input sample rate, excluding categorical variables
  • consultar_equipos_parque() and consultar_equipos_parque_no_agrupamientos() are now called consultar_equipos() and consultar_equipos_no_agrupamientos()
  • __iec61400_preparar_df_vacio_minutal() and it's aux methods are now more efficient
  • Added variables "Gen", "Con", "Pos" as boolean series for each equipment, into the datos_seg dataframe
  • Added columns "ENS_Lim","ENS_LimExt","ENS_LimInt","ENS_Ind","ENS_IndExt,"ENS_IndInt" to "consolidado" report
  • Added variable "Datos_%" which indicates the ammount of not null data per row, per element, into the datos_seg dataframe
  • Added variable "Registros" which indicates the ammout of rows (measured in time) on the excel file for that given element. Should alwasy equal 24
  • Fixed bug: .__procesar_d10s_5i() wasn't ensuring boolean dtypes for 'Gen','Con' and 'Pos', which impacted negatively on .elaborar_reporte_consolidado()
  • Fixed bug: .consolidar_todo() wasn't working with PEGARCIG
  • Fixed bug: .consolidar_todo() wasn't working adding up HOUR columns properly (the rest was fine)
  • Fixed bug: .consolidar_todo() wasn't finishing if there weren't any incidents for a given park on a given time window
  • Fixed bug: .explotar_incidencias() now works properly if there aren't any incidents on a given time window
  • Fixed bug: .exportar() was exporting empty dataframes, since they were not "None"
  • Fixed bug: Roll-up of availability parameters wasn't being done correctly. Now groupings are the mean or logic operator of their groupped elements.
  • Fixed bug: 10s pre-processing: Wind_dir_r was not being saved correctly for 'PLANT' element on self.datos_seg
  • Fixed bug: 10 second data is now being completed (in case of missing 10second records) with null values. This could have lead to serious problems when
  • Fixed bug: On many methods. From pandas >= 1.5.0 numpy series where linked and not copied when creating a new df. This created many bugs throughout the whole program.
  • unloading incidents to the 10 second data, since the main mechanism behind it is selection trhough .loc[t_stamp,:]
• TablasVC()
  • Added attributes .habilitaciones and .tipohabilitaciones which are new tables of "datos básicos". They contain COD related information of each plant
  • Added .consultar_cod(nemo_parque='',uc='') function, to quickly find out about a plant's COD (Birthday) date (grabs from the table above)
  • Added .consultar_año_op(fecha_actual,feca_cod) function. Returns the contractual year (Contractual age) of a power plant
  • Added .__crear_dict_tipo_empresa(). Which fills in thre internal attributes: _empresas_bop, _empresas_gen, _empresas_grid
  • Fixed bug: .consultar() would fetchdatos_basicos from offline file even when 'online' parameter was set. This was due to:
    • a logic error of variable db_off inside .consultar()
    • the order in which instructions were executed
  • Fixed bug: __procesar_incidencias_it2() wasn't executing properly when cargar_datos_basicos = 'offline' was set. Solved by calling the new function .__crear_dict_tipo_empresa() before business mapping
  • Fixed bug: Desconecting twice from DB when fetching incidencias
  • Fixed bug: __combinar_incidencias_locales_y_online() wasn't working properly. Duplicate dropping should only be done by 'ID'
  • Refactored __asignar_datos() .method, to make it modular per table

Version 0.0.16

• DatosCROM()
  • Added data invalidation when processing 10 second data. Now consecutive values of P, Q, Wind, Wind_dir and PPos != 0 and != max (for Ppos) are replaced with np.nan
  • Fixed bug when selecting no .parques ( [] ) and initializing a DatosCROM object
  • Fixed bug when truying to export specific reports through the generating functions (ie elaborar_curvas_de_potencia, interpretar_bajo_iec61400, etc.) and seeting the "exportar" parameter to True
  • Fixed bug. PI wasn't being correctly calculated. (* instead of / was being used)
  • Added "filtro_n" parameter to .elaborar_curvas_de_potencia(), to only show results with an n_count >= n per bin. Default 0 (deactivated)
  • Added feature. Ppos is being calculated if column doesn't exist, based on Curvas_de_potencia.xlsx on BLC's cloud under blc.get_dc_cfg() directory
  • "clientes" initializing parameter was removed (little use for it)
  • "solo_CROM" initializing parameter was removed (little use for it)
• TablasVC()
  • Improved the processing of incidents, now "Origin" is "NA" and np.nan resistent. Default = "INT"
  • Fixed bug through which a filter for discarded incidents wasn't working.
  • Refactored processing of incidents, it's more readable now

Version 0.0.15

• DatosCROM()
  • .exportar() was added. Helper function to export all loaded reports to xlsx format.
  • .explotar_incidencias() now takes in optional export parameters.
  • .incidencias_interpretar_bajo_iec61400() now takes in optional export parameters.
  • .incidencias_identificar_solapamientos() was added.
  • "curvas_de_potencia" parameter added while initializing, to force power curves to be computed automatically
  • .elaborar_curvas_de_potencia() is now more resistent to failures. Powercurves can be elaborated without having to process incidents before, although not recommended if user doesn't know what he is doing.
  • changed how __renombrar_lvl0() works, now it must be feed the equipment names.
• TablasVC()
  • fixed a bug, where .consultar_equipos_parque_no_agrupamientos would return "PLANT" twice for GRIOEO
  • "cargar_incidencias" and "cargar_datos_basicos" parameters added while initializing, with possible values being [Flase, True, 'offline', 'online']
  • now queries per default active incidents from the 1st day of two months prior to today (if True or 'online' values where passed to "cargar_incidencias" parameter). Then concatenates obtained results with offline results (if there are any).

Version 0.0.14

• TablasVC()
  • Fixed bug: Corrected functions that took equipments from blctools.TablasVC().conjuntogeneradores, now reading from .tipoequipo
  • Fixed bug: Now Parks without any grouping of it's generators can return the generators without grouping. Special Case: PEGARCIG
• SQLConnector()
  • now decrypts login data from BLC's secure fileserver.
• DatosCROM()
  • .consultar_equipos_parque() now retrieves the right elements if the plant doesn't havy any other grouping stages
  • .consultar_equipos_por_agrupamiento() now returns an empty list if the plant doesn't havy any other grouping stages
  • .rpt_consolidado(granularidad='1D') was added.
  • "forzar_reprocesamiento" parameter added while initializing, to force re-process 10 second data.
  • .rpt_curvas_de_potencia() was added
  • changed the behaviour of .fecha_i and .fecha_i setters. They now attempt to update "incidencias" and "archivos" depending on the values of fecha_i and fecha_f, but the setters dont try to order the dates automatically, which lead to an unwanted behaviour.
  • Changed completely how DatosCROM() now proceses raw data (10 seconds and Incidencias). It's more efficient, and the heavy lifting occurs while processing the 10-second data. Now index has 1 column (Time) and columns have a 3 level structure (p,e,v) = Park, Element, Variable

Version 0.0.13

• Created SQLConnector class to migrate to SQLAlchemy
• blctools.TablasVC now inherits from SQLConnector

Version 0.0.12

• Function blctools.fechas.obtener_periodo() was added (moved from blctools.ReporteBase())
• Added more parameter flexibility within all functions inside blctools.fechas.
• Objects PPO(), DTE() and DatosCROM() now accept the "periodo" initialization parameter (overrides fecha_i and fecha_f).
• bjects blctools.DatosCROM() and blctools.TablasVC() have now the "solo_CROM=False" initialization parameter.
• function blctools.TablasVC.consultar_nemoCammesa() has now the "solo_CROM=False" argument

Version 0.0.11

• Corrected blctools.PPO.periodo('mes_anterior')
• Function ReporteBase.a_excel() accepts now the parameter "exportar_consulta=bool"
• Objects PPO() and DTE() now accept the following initialization parameters
  • "filtro"
  • "periodo" (overrides fecha_i and fecha_f)
• No more unnecessary warnings on changing the "filtro" parameter
• Function blctools.fechas.mes_dia_1() and blctools.fechas.mes_ult_dia() now validate dates using blctools.fechas.validar_fechas()
• Functions blctools.fechas.validar_fecha() and .validar_fechas() now have a parameter "prevenir_futuro=True" as default, to prevent future dates or not.
• Fixed a bug, when changing "filtro" on functions of child classes of DTE() or PPO(), download and extraction directories wouldn't update properly.

Version 0.0.10

• Attribute blctools.DatosCROM.dir_salida sets to which directory the reports go out to (didn't have much impact before)
• Improved automatic Servicios directory search

Version 0.0.9

• Removed dependencies list
• Ability to download unfiltered PPOs/DTEs without specifying parks/plants list
• Incidents are no longer loaded by default on blctools.DatosCrom() objects

Version 0.0.8

• Corrected dependencies list

Version 0.0.6 ~ 0.0.7

• Bug fixes regarding file management
• Ability to customize filters when hitting CAMMESA's API
• PBA calculation according to IEC61400 is still functioning incorrectly

Version 0.0.5

• Most of the code is Object Oriented now.
• CAMMESA's forecasts have been added.

Version 0.0.2 ~ 0.0.4

• Fixed the install issues

Version 0.0.1

• Basic functionality is up and running

TO DOs

• Corroborar que la cantidad de grupos de columnas en los archivos 10 segundales, coincida con la cantidad de equipos del parque cuyo nombre de archivo enuncia.
• Convertir DatosCROM().parques a "nemos" automáticamente.
• Mejorar los mensajes de procesamiento nivel 0, 1 y 2 en todas las funciones
• Fix Bug: Code could break by passing an '' element into parques initialization parameter