[v9.0.0a28] *WorkloadManagementSystem FIX: (#7574) serverUtils: jit imports FIX: (#7534) Pilots submitted by SiteDirector won't add the pythonVersion flag FIX: (#7521) Fix memory reporting FIX: (#7510) SandboxStore: add VO if needed *WorkloadManagement FIX: (#7571) support file:/... as a location for the pilot files FIX: (#7564) make sure CVMFS_locations is a list CHANGE: (#7553) Remove files from the RemoteRunner execution FIX: (#7552) JobCleaningAgent: fix exception in deleteJobsByStatus caused by mismatching job ID types NEW: (#7529) introduce JobWrapperOfflineTemplate for uses in systems without external connectivity CHANGE: (#7460) introduce JobWrapper.preprocess, process and postprocess *test NEW: (#7570) added pilot workflow tests to integration_tests *Core FIX: (#7569) Support M2Crypto 0.40.0+ CHANGE: (#7566) Replace the default PFN type ROOT_All with ROOT FIX: (#7524) Depend on packaging *environment.yml NEW: (#7555) add cwltool *Resources FIX: (#7545) TimeLeft utility was unable to get values from the cfg FIX: (#7532) support the case where HTCondor kills the jobs *Test FIX: (#7540) Redirect the output of popen in a file to fix #7473 FIX: (#7539) extra_module default to empty list *FrameworkSystem CHANGE: (#7511) ProxyDB: removed tables ProxyDB_Proxies and ProxyDB_Tokens [v9.0.0a27] *Core FIX: (#7505) plotting TypeLoader works with editable installation NEW: (#7453) Introduce an RPC stub equivalent for DiracX *Test CHANGE: (#7502) use pytest-rerun instead of flaky *WorkloadManagementSystem CHANGE: (#7498) removed GridEnv FIX: (#7497) If the SoftwareDistModule is set in the Operations Section, add it to the Job JDL to restore previous behaviour NEW: (#7453) FutureJobStateUpdate.setJobStatusBulk return a DiracX RPC stub *WorkloadManagement FIX: (#7493) check the VO from the task queues before submitting pilots FIX: (#7488) JobAgent.setupProxy takes owner instead of ownerDN *RequestManagementSystem CHANGE: (#7453) adapt ForwardDISET to DiracX stub *FrameworkSystem CHANGE: (#7442) removed NotificationDN [v9.0.0a26] *FrameworkSystem FIX: (#7491) dirac-proxy-info without the dirac group *WorkloadManagement FIX: (#7490) missing result in return statement *tests NEW: (#7484) add the state key in the test environment to fix diracx execution [v9.0.0a25] *Accounting FIX: (#7486) Fix errors during insert into ac_in_* tables. *Core FIX: (#7483) Fix DISET calls with proxy to be used passed as an argument *RequestManagementSystem FIX: (#7482) RequestTask - download no-VOMS proxy if the owner group does not define VOMSRole [v9.0.0a24] *WorkloadManagement FIX: (#7480) JobCleaningAgent - select random jobs for deletion rather than head and tail jobs. FIX: (#7475) JobID type in PushJobAgent *Interfaces CHANGE: (#7472) dfind - more explicit failure report *DataManagementSystem NEW: (#7471) add tools for consistency checks *tests FIX: (#7470) check running containers in integration tests script *WorkloadManagementSystem CHANGE: (#7464) PilotAgentsDB: removed OutputReady and Broker fields [v9.0.0a23] *WorkloadManagement FIX: (#7458) jobID type issue in JobAgent *ConfigurationSystem FIX: (#7454) getQueue() overriding the CE tags *FrameworkSystem FIX: (#7451) dirac-proxy-init printInfo without the dirac group [v9.0.0a22] *WorkloadManagement FIX: (#7448) get pilot logging info with a token from an AREXCE *WorkloadManagementSystem FIX: (#7447) Each job has its own JobReport in JobAgent FIX: (#7447) JobAgent exits when all the jobs have been processed FIX: (#7446) StatesAccountingAgent: skip the first iteration in order to avoid double commit after a restart CHANGE: (#7439) SandboxStore: remove external SE feature FIX: (#7436) JobDB: fix mismatch of string and integer jobIDs. In some cases API calls would fail because JObDB.getJobParameters and JobDB.getJobsAttributes return dictionaries with integer keys, while the function was called with a string jobID. This fixes for example the StalledJobAgent being unable to reschedule matched jobs. NEW: (#7425) TornadoPilotLoggingHandler modify the handler to accept VO name sent by a pilot. Required in a case where the VO cannot be guessed from a proxy. This change requires https://github.com/DIRACGrid/Pilot/pull/230 NEW: (#7421) SandboxDB: add VO field CHANGE: (#7414) move the content of SubmissionPolicy in SiteDirector *FrameworkSystem FIX: (#7443) correctly set the duration of tokens in cache *RequestManagementSystem FIX: (#7441) make sure OwnerDN is defined before trying to access its value *MonitoringSystem CHANGE: (#7432) moved to weekly indices for agent and service monitoring [v9.0.0a21] *Test CHANGE: (#7417) write the diracx CsSync config in the CS instead of taking the yaml file from diracx repo *FrameworkSystem CHANGE: (#7413) removed Proxies persistency flag *Core FIX: (#7412) Adding VOMS extensions without having environment variables set NEW: (#7412) Add DIRAC_DISABLE_GCONFIG_REFRESH environment variable to prevent gConfig being accidentally used FIX: (#7409) Use proxy lifetime for tokens from legacy proxy exchange (https://github.com/DIRACGrid/diracx/issues/130) *WorkloadManagement FIX: (#7409) Add DiracX to payload proxies used by compute elements (#7402) FIX: (#7406) SiteDirector should not interact with CEs if there is 0 pilot to submit *WorkloadManagementSystem CHANGE: (#7407) JobDB simplifications CHANGE: (#7405) Removed Private Pilot functionality [v9.0.0a20] *WorkloadManagement FIX: (#7399) JobAgent rescheduling wrong jobs FIX: (#7387) JobAgent interaction with JobMonitoringClient *WorkloadManagementSystem CHANGE: (#7396) PilotAgentsDB: move from OwnerGroup to VO FIX: (#7379) The callback for the Stager was failing, because of a type mismatch in the jobID used to retrieve the status. Jobs never came out of Staging. NEW: (#7375) pilotWrapper: using CVMFS_locations for discovering the pilot files *FrameworkSystem FIX: (#7391) send notifications for expiring proxies Documentation on how to deploy a third party tool (fluent-bit) to grab, format and send Dirac current logs to ElasticSearch and/or splitted logs files *Resources FIX: (#7376) AREXCE should break when a valid delegation ID is found *Core FIX: (#7374) Converting p12 files with filenames containing special characters [v9.0.0a19] [v9.0.0a18] [v9.0.0a17] [v9.0.0a16] [v9.0.0a14] [v9.0.0a14] [v9.0.0a15] [v9.0.0a14] [v9.0.0a13] [v9.0.0a12] [v9.0.0a11] [v9.0.0a10] [v9.0.0a9] [v9.0.0a8] [v9.0.0a7] *Resources FIX: (#7366) make sure the WLCG accounting file is json [v9.0.0a6] *WorkloadManagementSystem FIX: (#7364) added SiteDirector option for CVMFS_locations [v9.0.0a5] *FrameworkSystem FIX: (#7360) wrong service uptime calculation *Resources FIX: (#7357) get batch system details from local cfg instead of environment variables FIX: (#7349) remove valid argument from setToken *docs NEW: (#7356) added doc for PreInstalledPilotEnv *Core NEW: (#7344) Introduce DIRAC_MYSQL_CONNECTION_GRACE_TIME to specify the grace time of the MySQL connection pool *TransformationSystem NEW: (#7337) add an index on Status and Type for the TransformationDB.Transformations table *WorkloadManagementSystem FIX: (#7332) basic JobMonitoring client fixes FIX: (#7331) JobAgents will set jobStatus=Failed/Payload failed if and only if the job was previously Running *CORE FIX: (#7332) allow for proxyLocation to work with host certificate [v9.0.0a4] *docs FIX: (#7330) just use `pip install DIRAC` for DIRAC client install FIX: (#7330) updated picture for py3 stack *Interfaces FIX: (#7328) Improve performance of job delete/kill/reschedule API FIX: (#7322) Lower log level for printing version in submitJob *WorkloadManagementSystem FIX: (#7327) Allow "ANY" in the valid sites list NEW: (#7318) initial FutureClient for JobStateUpdate FIX: (#7316) ElasticJobParametersDB: do not configure the IndexPrefix name [v9.0.0a3] *WorkloadManagement FIX: (#7305) Use copy+remove in PilotSync agent to avoid SELinux problems FIX: (#7288) WMSUtilities supports VO-specific token tags *WorkloadManagementSystem FIX: (#7301) correctly handle UTC in JobLoggingDB float timestamp *Core FIX: (#7297) convert exception object to string representation in ElasticSearchDB *DataManagementSystem CHANGE: (#7297) default proxy lifetime for FTS is 36h and force the redelegation when 12h are left *FrameworkSystem FIX: (#7284) disabledVOs not available when doing dirac-configure *DiracX NEW: (#7261) DiracX credentials are now included in proxy PEM files created by DIRAC NEW: (#7261) DiracX is now mandatory 🎉 [v9.0.0a2] *WorkloadManagementSystem FIX: (#7282) JobStatus: allow WAITING -> KILLED FIX: (#7263) SiteDirector acts for a VO *FrameworkSystem FIX: (#7280) show status SysAdminCLI KeyError *Resources FIX: (#7279) getting more details about failed/aborted pilots from HTCondor *Core FIX: (#7273) createClient: also look at HandlerMixin classes to find `export_`ed functions. Fixes the client Documentation creation, fixes #7265 *WorkloadManagement NEW: (#7271) SiteDirector - added preinstalledEnv and preinstalledEnvPrefix pilot options [v9.0.0a1] *Core NEW: (#7258) Support using DiracX as a backend for RPC calls *WorkloadManagementSystem CHANGE: (#7257) Refactor PilotLoggingAgent: download proxies at initialisation. Renew a proxy if it is about to expire. *Resources NEW: (#7256) Add support for loading external providers in CloudCE NEW: (#7256) Add OpenNebula6 (XMLRPC) provider for CloudCE/libcloud *Subsystem FIX: (#7253) prevent AREXCE from creationg delegations each time a CE got a small and temporary issue *docs NEW: (#7252) add documentation about setting up DIRAC to submit pilots with tokens *WorkloadManagement CHANGE: (#7250) pilot submission with tokens in a multi-VO context [v8.1.0a23] *TransformationSystem NEW: (#7239) TransformationCleaningAgent: Use the RequestManagementSystem to asynchronously remove files, rather than serially in the agent run [v8.1.0a22] *WorkloadManagementSystem FIX: (#7238) Output sandbox are assigned when using DiracX CHANGE: (#7237) Removed interaction with MJF FIX: (#7226) SiteDirector adds wnVO Pilot options, if needed CHANGE: (#7215) add VO info in the table Jobs of JobDB *WorkloadManagement FIX: (#7234) AREX does not return an error if delegation not found FIX: (#7230) Group to vo conversion in TaskQueueDB *ConfigurationSystem CHANGE: (#7225) explicitly state the VO in each default group upon installation *DiracX CHANGE: (#7222) Get a access/refresh token pair when generating a proxy NEW: (#7222) Support using diracx as backend for the SandboxStoreHandler [v8.1.0a21] *Core FIX: (#7216) replace ColorBar.draw_all() with figure.draw_without_rendering() *WorkloadManagement FIX: (#7213) emove OwnerDN from SiteDirector & PilotStatus *WorkloadManagementSystem CHANGE: (#7190) add VO info in TaskQueueDB FIX: (#7187) code simplification for SandboxStoreClient [v8.1.0a20] *Test FIX: (#7206) manually start the extra services when running the integration test NEW: (#7143) allow integration tests to run against DiracX *TransformationSystem CHANGE: (#7204) TransformationCleaningAgent removes files by chunk FIX: (#7200) Drop TS permission check for all read-only functions *WorkloadManagement FIX: (#7201) JobMonitoringClient uses the new DiracClient class CHANGE: (#7194) Job.setNumberOfProcessors raises an exception if the function arguments aren't valid. Please follow the template: *WorkloadManagementSystem FIX: (#7196) do not try to insert nan values in AccountingDB NEW: (#7143) call the diracx Job Monitoring endpoint NEW: (#7105) Enhance dirac-admin-get-pilot-output to download remote pilot logs. Currently only from a SE. *FrameworkSystem FIX: (#7192) replace create_access_token with create_token *ConfigurationSystem CHANGE: (#7143) do not look for the shifter under the Default or section *Core NEW: (#7143) get a token for a proxy [v8.1.0a19] *ConfigurationSystem CHANGE: (#7182) VOMS2CS agent only sends a mail in case of change in the CS *WorkloadManagementSystem FIX: (#7180) passing pilotDN instead of owner in getPilotProxyFromVOMSGroup() [v8.1.0a18] *Production CHANGE: (#7162) Move from DN to Username *WorkloadManagementSystem CHANGE: (#7157) PilotAgentsDB: move DN in favor of username [v8.1.0a17] *TransformationSystem NEW: (#7170) Implement a finer grained permissions model for Transformations. *DataManagementSystem FIX: (#7168) better error reporting when an existing LFN has no replicas (at least it doesn't crash) *WorkloadManagement CHANGE: (#7167) Stop using PID namespaces with SingularityCE *Resources FIX: (#7160) default providerType if not found in IdProviderFactory *WorkloadManagementSystem FIX: (#7155) Adding Owner as JDL requirement CHANGE: (#7151) factorize some utilities of JobSubmit into functions [v8.1.0a16] *FrameworkSystem FIX: (#7149) Do not overwrite the tornado port if already set FIX: (#7117) replaced deprecated importlib read_text function to fix failure in CI tests *Resources FIX: (#7147) MultiVO proxy renewal in AREX FIX: (#7146) Propagate environment variables from host to container. FIX: (#7134) Don't print client_secret in oauth2 debug FEAT: (#7120) allow to pass credentials when downloading the SRR file CHANGE: (#7103) ARC and ARC6 are now deprecated CHANGE/FIX: (#7102) proxy renewal logic in AREX FIX: (#7102) submitting tokens with AREX FIX: (#7102) correctly report aborted pilots with AREX *TransformationSystem FIX: (#7144) dirac_production_runjoblocal: updated list of Pilot modules CHANGE: (#7124) use userName instead of DN *Core FIX: (#7142) look also for DIRAC.rootPath/etc/dirac.cfg before printing a warning of not found cfg *docs change: Replace the use of sphinx-panels with sphinx-design, this unblocks the use of latest sphinx versions (panels was requiring sphinx < 5) FIX: (#7133) formatting item list *WorkloadManagement NEW: (#7132) Introducing 'Scouting' status in job state transitions *ResourceStatusSystem FIX: (#7120) Add the Error status in the scripts *WorkloadManagementSystem FIX: (#7101) JobManagerHandler: get the OwnerDN only when needed. [v8.1.0a15] *Resources FIX: (#7100) concatenate tags instead of overriding them in ComputingElement.initializeParameters() NEW: (#7079) audienceName attribute in CEs NEW: (#7065) add a test for the PoolCE to highlight the fact that submission failures cannot be handled by the caller with the return value *WorkloadManagementSystem FIX: (#7098) Exceptions in StalledJobAgent CHANGE: (#7097) modifying access specifiers of JobPolicy from private to protected FIX: (#7081) disable activityMonitoring for JobAgent NEW: (#7079) audience support in SiteDirector CHANGE: (#6973) validating the JDL job format using a pydantic model *FrameworkSystem FIX: (#7094) Support Authlib v1.2.1 *FrameworkSyste FIX: (#7079) fix TokenManager(Client/Handler/Utilities) *WorkloadManagent NEW: (#7068) New TornadoSandboxStoreHandler implementation *Core NEW: (#7068) TornadoClient - added sendFile() method FIX: (#7067) Close DISET selector when we no longer need it *WorkloadManagement FIX: (#7065) management of the job status in JobAgent NEW: (#7065) add a test for the JobAgent to make sure the status of the submissions are correctly handled *docs NEW: (#7065) documentation about ComputingElement *tests NEW: (#7024) added script for comdirac hackathon certification tests [v8.1.0a14] *WorkloadManagementSystem NEW: (#7059) new script dirac-admin-add-pilot *dashboard NEW: (#7039) add missing dashboards of issue #6742 also to the LHCb grafana organisation. The dashboards can be checked/reviewed at https://monit-grafana.cern.ch/dashboards/f/qwdBd3KVk/dev-dirac-certification Please follow the template: *docs FIX: (#7023) added remaining info from COMDIRAC Wiki. Addresses part of #6746 [v8.1.0a13] *Core FIX: (#7058) print a warning message when no CFG file is found FIX: (#6976) retry upon DNS lookup error FIX: (#6955) HandlerManager - create separate class for each service handler FIX: (#6955) BaseRequestHandler - fullComponentName can be only defined in the configuration *docs FIX: (#7057) added a note about removal of elastic job parameters CHANGE: (#6969) more details in the "supercomputers" section *Resources CHANGE: (#7055) use double slash in Echo xroot URLs FIX: (#7042) reduce memory consumption of AREXCE.getJobOutput() CHANGE: (#7036) remove CREAM CE FIX: (#7028) SSHCE getPilotOutput remove stamp FIX: (#7000) ARCLogLevel FIX: (#6999) remove executables definition from AREXCE.submitJob() NEW: (#6969) cleanJob() in AREX CHANGE: (#6969) executables from inputs are passed as executables in the XRSL in ARC and AREX *test FIX: (#7053) Workflow tests need pilot.cfg to work *WorkloadManagementSystem FIX: (#7052) Implementation of DefaultLocalCEType option FIX: (#7046) fix(StalledJobAgent): after a job was Killed, we have to force the Failed. CHANGE: (#7033) TokenManagerHandler provides client access token if username is not provided. CHANGE: (#7031) Using JobManager instead of JobDB for rescheduling stalled jobs CHANGE: (#7019) removed old format of ElasticJobParametersDB NEW: (#7017) add the possibility of adding generic options for running the pilots FIX: (#6980) JobStatus: add staging to checking transition, needed by StorageManagementSystem (Stager) NEW: (#6969) tests for RemoteRunner CHANGE: (#6969) RemoteRunner can deal with MP applications, clean job outputs and process more applications FIX: (#6969) Modify the DIRACVersion in the PushJobAgent *WorkloadManagement NEW: (#7042) md5 checksum comparison in RemoteRunner NEW: (#7041) Add /Resources/Computing/DefaultLocalCEType option FIX: (#7007) return specific exit code in RemoteRunner in case of failure FIX: (#7006) Getting the value of /LocalSite/RemoteExecution *RequestManagementSystem FIX: (#7013) fix the monitoring of Operations not yet entered into the DB *FrameworkSystem CHANGE: (#6967) enable/disableLogsFromExternalLibs only used from LocalConfiguration FIX: (#6967) remove a deprecated method CHANGE: (#6967) gLogger date format NEW: (#6967) type hinting in gLogger *Interfaces FIX: (#6961) Removed dinit, sessions are now instantiated automatically. Also removed DCOMMANDS_PPID. PR for adding Dirac side code related to https://github.com/DIRACGrid/WebAppDIRAC/pull/729/commits CHANGE: (#6956) Add getTokensByUserID method and make it compliant with ProxyManager code *Integration tests FIX: (#6954) Make error nicer when the IAM docker instance is not working properly *dashboards NEW: (#6911) added json export of Grafana demo dashboard [v8.1.0a12] *WorkloadManagementSystem FIX: (#6953) Watchdog fix: always report correct types FIX: (#6931) make singletons of VirtualMachineDB and PilotAgentsDB PR for fixing dead link in doc (monitoring part) FIX: (#6949) change urls for WMS and Logs dashboards *tests CHANGE: (#6948) Integration tests: upped Opensearch version to 2.1.0 NEW: (#6937) integration tests for IdProviders *Core CHANGE: (#6947) remove support for MySQL 5.7 FIX: (#6944) protection for opensearch_dsl being merged in opensearchpy FIX: (#6924) cast method names to string before sending monitoring *Resources NEW: (#6937) unit tests for IdProviders FIX: (#6927) HTCondorCEComputingElement - set _CONDOR_AUTH_SSL_CLIENT_CADIR before executing condor commands *Tornado FIX: (#6933) do not use activityMonitor in TornadoREST *FrameworkSystem FIX: (#6922) stdoutJson logger prints all the necessary fields FIX: (#6922) MicrosecondJsonFormatter does not dump DIRAC specific fields (`timeStampIsShown`, `colorIsShown`, etc) [v8.1.0a11] *Resources FIX: (#6913) better handle exceptions in AREXCE CHANGE: (#6902) (AREX)ComputingElement - use proxy only if it is set up FIX: (#6883) add workaround for https timeout when dowloading file *WorkloadManagementSystem FIX: (#6913) config helper error message FIX: (#6907) SiteDirector: failure in submitting Accounting or Monitoring report only prints an error *MonitoringSystem FIX: (#6904) while updating ES documents, wait and retry on ConflictError CHANGE: (#6875) introduce gMonitoringDB global instance *Interfaces FIX: (#6903) make -F (file) option work for drm command. This fixes #6896 *WorkloadManagement NEW: (#6902) Enable PilotManager to manage pilots with tokens *Core FIX: (#6901) Tornado PeriodicCallbacks are actually called FEAT: (#6901) Tornado Handlers report to ActivityMonitor FEAT: (#6901) Activity Monitoring reports useful information FIX: (#6874) TornadoREST - fix the error in resolution of exposed methods Thank you for writing the text to appear in the release notes. It will show up exactly as it appears between the two bold lines Please follow the template: *Production System CHANGE: (#6898) change column types in ProductionDB from VARCHAR(255) to LONGTEXT *FrameworkSystem CHANGE: (#6897) Default log level is INFO NEW: (#6897) Define global configuration options for log level *ConfigurationSystem FIX: (#6888) Fix the connection to ES database when using certificates *DataManagementSystem CHANGE: (#6884) refactor (DFC): rewrite the file/replica deletion stored procedure as they were in 5.6 CHANGE: (#6880) FTS3DB uses `in` instead of `not in` for some query CHANGE: (#6880) FTS3DB removal of operations uses python datetime instead of mysql builtin [v8.1.0a11] *Resources FIX: (#6913) better handle exceptions in AREXCE CHANGE: (#6902) (AREX)ComputingElement - use proxy only if it is set up FIX: (#6883) add workaround for https timeout when dowloading file *WorkloadManagementSystem FIX: (#6913) config helper error message FIX: (#6907) SiteDirector: failure in submitting Accounting or Monitoring report only prints an error *MonitoringSystem FIX: (#6904) while updating ES documents, wait and retry on ConflictError CHANGE: (#6875) introduce gMonitoringDB global instance *Interfaces FIX: (#6903) make -F (file) option work for drm command. This fixes #6896 *WorkloadManagement NEW: (#6902) Enable PilotManager to manage pilots with tokens *Core FIX: (#6901) Tornado PeriodicCallbacks are actually called FEAT: (#6901) Tornado Handlers report to ActivityMonitor FEAT: (#6901) Activity Monitoring reports useful information FIX: (#6874) TornadoREST - fix the error in resolution of exposed methods Thank you for writing the text to appear in the release notes. It will show up exactly as it appears between the two bold lines Please follow the template: *Production System CHANGE: (#6898) change column types in ProductionDB from VARCHAR(255) to LONGTEXT *FrameworkSystem CHANGE: (#6897) Default log level is INFO NEW: (#6897) Define global configuration options for log level *ConfigurationSystem FIX: (#6888) Fix the connection to ES database when using certificates *DataManagementSystem CHANGE: (#6884) refactor (DFC): rewrite the file/replica deletion stored procedure as they were in 5.6 CHANGE: (#6880) FTS3DB uses `in` instead of `not in` for some query CHANGE: (#6880) FTS3DB removal of operations uses python datetime instead of mysql builtin [v8.1.0a10] *Tests CHANGE: (#6861) factorize FTS3 tests [v8.1.0a9] *ConfigurationSystem FIX: (#6868) VOMS2CSAgent: strip whitespaces from DN entried and nickname NEW: (#6865) Resources helper - getQueues() - can select queues by Tag values also *FrameworkSystem FIX: (#6865) TokenManager - use refreshToken flow to generate access tokens from the stored refresh tokens *WorkloadManagement NEW: (#6865) SiteDirector enabled to select queues by tags NEW: (#6865) SiteDirector sets up tokens for ComputingElements configured with the Token tag *Resources NEW: (#6865) HTCondorComputingElement, ARC(6)ComputingElement and AREXComputingElement enabled to for job operations with tokens FIX: (#6842) Complete AREX renewal *DataManagementSystem NEW: (#6863) ReplicateAndRegister FTS mode also checks the SE status NEW: (#6863) add a delay in monitoring in the FTS3Agent NEW: (#6860) dirac-dms-filecatalog-cli - use the default VO FileCatalog if not explicitly specified CHANGE: (#6835) Speed up ReplicateAndRegister operation *dashboards/PilotSubmissions FIX: (#6858) efficiency plots now use heatmaps FIX: (#6858) index patterns are now environment agnostic *dashboards/ServiceMonitoring FIX: (#6858) index patterns are now environment agnostic *dashboards/AgentMonitoring FIX: (#6858) index patterns are now environment agnostic *dashboards/PilotsHistory FIX: (#6858) style of plots FIX: (#6858) index patterns are now environment agnostic *Interfaces NEW: (#6853) Add setPriority to Job API FIX: (#6849) removed print statements and replaced with gLogger as appropriate as requested in #6746 *RequestManagementSystem CHANGE: (#6848) ReqProxy does not attempt a direct forward to central ReqManager [v8.1.0a8] *Resources FIX: (#6832) Update AREX delegation renewal FIX: (#6830) Fix AREX ARCRESTTimeout FIX: (#6817) fix account parameter in SSHBatchCE *TransformationSystem FIX: (#6829) keep transID an int in utilities *DataManagementSystem FIX: (#6819) specify the same column in order_by and distinct in FTS3DB.getNonFinishedOperations *Tests NEW: (#6812) Integration test for StorageElementHandler and FileHelper [v8.1.0a7] *Core FIX: (#6811) don't use matplotlib deprecated methods FIX: (#6793) Fix exception in FileHelper.networkToString on Python3 FIX: (#6784) Tornado components are initialized using the Tornado system section FIX: (#6784) each Tornado handler reads its own log level New: dirac-configure warns you about existing config file FIX: (#6731) Fix in FileHelper module writing to file, fixed exception "TypeError: write() argument must be str, not bytes, because file was opened in the wrong mode. CHANGE: (#6706) debug log level instead of info in a few places *Resources NEW: (#6806) [ALL]ComputingElements - pass pilot stamp to the pilot environment FIX: (#6768) do not handle ECOMM when removing file FIX: (#6753) Add a note about XRSL times in ARC(6) vs AREX NEW: (#6729) SensitiveDataFilter in LogFilters FIX: (#6719) remove stupid CS sync when getting MQ config CHANGE: (#6690) drop the WholeNode CE option from Condor CHANGE: (#6683) https comes before gsiftp in the preference list NEW: (#6683) Singularity detects locally mounted file system on the worker node to bind mount FIX: (#6681) fix upload from relative path *Docs FIX: (#6801) Removed Google forum from docs NEW: (#6786) Improve documentation on installing HTTPs services FIX: (#6732) nginx takes aliases into account NEW: (#6683) document how to access locally mounted file system *DataManagementSystem CHANGE: (#6787) take into account all possible sources for intermediate copy when replicating files CHANGE: (#6763) Removed deprecated StorageElementProxy and FileCatalogProxy NEW: (#6759) Provide `diskOnly` option in getFile. Defaul is `False` CHANGE: (#6752) adapt FTS3DB to sqlalchemy 2.0 NEW: (#6697) Introduce getDirectoryDump method in the DFC CHANGE: (#6697) dirac-dms-user-lfns uses getDirectoryDump method CHANGE: (#6697) DataManager getReplicasFromDirectory and __cleanDirectory use getDirectoryDump FIX: (#6696) fix a non initialized variable when doing a TPC with gfal2 FIX: (#6696) propagate error number when calling getTURL in SRM2 FIX : (#6696) ConsistencyInspector default value should be a dict and not a list *ConfigurationSystem FIX: (#6779) VOMS2CSSynchronizer: strip leading and trailing whitespaces from DN entries FIX: (#6706) getMonitoringBackends returns a list *WorkloadManagement FIX: (#6756) Repect paramList argument in ElasticJobParametersDB.getJobParameters FIX: (#6749) Reorder pilot downloads to minimise race condition FIX: (#6704) Calculating index for JobParametersDB for string job ids *Interfaces NEW: (#6755) Merge COMDIRAC *RequestManagementSystem CHANGE: (#6752) adapt ReqDB to sqlalchemy 2.0 *FrameworkSystem CHANGE: (#6743) Increased length of MySQL field UserProfileDB.up_Users to 64 (was 32) CHANGE: (#6729) Refactor access to log filters and Logging.__init__() FIX: (#6718) ComponentSupervisionAgent: Fix problem if two instances in different systems but with the same name are present on the same machine (e.g., Framework/Monitoring and Monitoring/Monitoring) CHANGE: (#6686) Removed old RabbitMQ specific interface *WorkloadManagementSystem FIX: (#6706) Matcher handles cases of empty Tag in resource description FIX: (#6684) InputData optimizer: keep the tape LFNs in *Accounting FIX: (#6695) JobsPerPilot plot when no data available [v8.1.0a6] *Resources FIX: (#6620) convert ARC to DIRAC JobID in AREXCE FIX: (#6618) SSHBatchComputingElement needs self.account FIX: (#6608) HTCondorCE: use CS location Resources/Computing/CEDefaults/HTCondorCE instead of Resources/Computing/HTCondorCE FIX: (#6595) fix delegation process in the AREXCE *WorkloadManagementSystem FIX: (#6618) encoding proxy in PilotWrapper.py FIX: (#6609) ExtraPilotOptions can be a comma-separated list *StorageManagamentSystem FIX: (#6617) fix py3 incompatibility in wakeupOldRequests *DataManagementSystem FIX: (#6591) Faster query to list directory for DFC LHCb managers [v8.1.0a5] *Resources FIX: (#6577) multi-node allocations with SLURM FIX: (#6558) AREXCE._getDelegation() returns an error if the process cannot be completed *ConfigurationSystem NEW: (#6576) BDII2CSAgent: allow using AREX or ARC6 Computing Elements instead of ARC, fixes #6541 *tests NEW: (#6569) added system/rms_script.sh test for basic RMS test *WorkloadManagementSystem CHANGE: (#6568) improve killing of process with psutil CHANGE: (#6556) completely removed old PilotsLogging machinery *DataManagementSystem NEW: (#6567) TornadoS3GatewayHandler service *ProductionSystem NEW: (#6567) TornadoProductionManagerHandler service *StorageManagementSystem NEW: (#6567) TornadoStorageManagerHandler service *MonitoringSystem CHANGE: (#6563) remove setup *Subsystem CHANGE: (#6561) Swapped the default of EnableSecurityLogging flag FIX: (#6553) simplified NotificationHandler *Core CHANGE: (#6466) removed DIRAC_USE_JSON_DECODE, flipped DIRAC_USE_JSON_ENCODE [v8.1.0a4] *Resources NEW: (#6530) add the ARCLogLevel option in ARC NEW: (#6530) add the ComputingInfoEndpoint in ARC6 FIX: (#6530) restore a commented line in AREX._reset() FIX: (#6524) Support using ARC that was compiled against SWIG 4.1+ NEW: (#6506) Added network selection to CloudCE. FIX: (#6505) simplifying `if` statements for setting bools *WorkloadManagement FIX: (#6525) Ensure cpuNormalizationFactor is a float FIX: (#6489) pass CE parameters to dirac-jobexec instead of writing them in the configuration *WorkloadManagementSystem FIX: (#6521) JobDB.getJobAttributes() returns an empty dict if jobID does not exist FIX: (#6479) reduce the PollingTime of JobAgent to 20s *ResourceStatusSystem FIX: (#6513) GOCDB parser to understand also real URLs FIX: (#6501) replace pre-Python 2.5 ternary syntax *docs FIX: (#6512) Added CloudComputingElement to AdminGuide/Resources *ConfigurationSystem FIX: (#6505) simplifying `if` statements for returning bools *DataManagementSystem FIX: (#6505) simplifying `if` statements for returning bools FIX: (#6505) use `not in` instead of `not ... in` FIX: (#6505) simplify loop with `any` call *FrameworkSystem FIX: (#6505) simplifying `if` statements for setting bools FIX: (#6505) simplified if statement for checking list/string len FIX: (#6501) replace pre-Python 2.5 ternary syntax NEW: (#6450) added TornadoComponentMonitoringHandler NEW: (#6450) added TornadoNotificationHandler NEW: (#6450) added TornadoUserProfileManagerHandler *TransformationSystem FIX: (#6505) simplifying `if` statements [unneeded `not`] FIX: (#6505) simplified if statement for checking list/string len *Core FIX: (#6480) quoting the DB passwords for special characters FIX: (#6462) Scripts: adapt to proper way of getting console_scripts entry_points, mandatory for importlib_metadata v5 FIX: (#6459) Setting timeout when adding VOMS extension *tests NEW: (#6463) added Glasgow to sites that should work(TM) in multiVO test NEW: (#6454) launching the Optimizers in integration-tests *Docs FIX: (#6462) Fail if one the commands cannot be documented [v8.1.0a3] *Docs FIX: (#6457) Fix the title for the Command Reference Page, fixes #6455 *Interfaces CHANGE: (#6452) remove JobRepository functionality *FrameworkSystem CHANGE: (#6451) remove support for myProxy *Core FIX: (#6448) Only use X- headers if /WebApp/Balancer is defined FIX: (#6428) MySQL tracer bugs *ConfigurationSystem FIX: (#6444) remove warning logs when initializing ElasticDB parameters *WorkloadManagementSystem FIX: (#6441) fix the timestamp of initial logging records FIX: (#6436) WMSUtilities.killPilotsInQueues have a consistent return value FIX: (#6432) removed unused SubmissionMode CE parameter NEW: (#6429) Information about VirtualOrganization parameter added. *Test FIX: (#6428) server installation do not install JobAgent *MonitoringSystem CHANGE: (#6426) Remove PilotSubmissionPlotter *WorkloadManagement CHANGE: (#6380) Remove VMDIRAC components [v8.1.0a2] *DataManagementSystem FIX: (#6414) Proper casting for data sent to the monitoring NEW: (#6410) backport getDestinationSEList from LHCbDIRAC FIX: (#6392) DirectoryLevelTree: fix missing keyword argument name causing failures to upload files for example *Tests NEW: (#6410) introduce generateDIRACConfig utility to generate DIRAC config during tests *Docs CHANGE: (#6410) move site documentation from ConfigRef to Resources and dirac.cfg NEW: (#6410) document Countries *WorkloadManagementSystem CHANGE: (#6410) JobWrapper uses getDestinationSEList rather that resolveSEGroup CHANGE: (#6408) ElasticJobParametersDB: create a new index every 1M jobs FIX: (#6397) JobScheduling: select the correct RescheduleDelay instead of 1 higher *MonitoringSystem CHANGE: (#6409) Remove DataOperationPlotter *Production NEW: (#6406) add dirac-prod-complete command *Core FIX: (#6398) Use safer mode for grid-security directories FIX: (#6397) TimeUtilities.fromString: when given a datetime.datetime, return the same object, instead of None. Also Fixes JobScheduling issue [v8.1.0a1] *WorkloadManagementSystem FIX: (#6388) replace opPath by opChain in JobPath *DataManagementSystem FIX: (#6385) Monitoring does not update the dict using by the accounting, avoiding a crash NEW: (#6385) FailedDataOperation monitoring now keeps track of interactive/job failures NEW: (#6383) cancel the FTS transfers when an RMS request is found to be canceled *Core FIX: (#6379) Explicitly depend on db12 CHANGE: (#6337) Use Python's default SSL ciphers by default *Resources CHANGE: (#6378) negative free space value is transformed to 0 [v8.0.0] *Resources FIX: (#6359) new directory creation specifies the permission in octal mode NEW: (#6357) Add a setGfalSetting decorator CHANGE: (#6357) add a 30s timeout for removal at RAL *WorkloadManagementSystem NEW: (#6355) The default InputDataDirectory used by DownloadInputData can be overridden by setting /LocalSite/InputDataDirectory FIX: (#6354) Stop instance from VirtualMachineMonitor [v8.0.0a29] *Core NEW: (#6347) Introduce ``DIRAC_MYSQL_OPTIMIZER_TRACES_PATH`` environment variable for advance optimization of MySQL calls CHANGE: (#6347) calls to the MySQL class with keyword parameters MUST now be named FIX: (#6345) Setting codeProperties["version"] in AgentModule [v8.0.0a28] *Core FIX: (#6341) createClient now creates subclasses of the original class NEW: (#6335) DIRAC.initialize now provides keyword arguments to customise initialisation NEW: (#6335) SecurityProperty enum for expressing security properties which can be assigned to groups/hosts NEW: (#6335) LogLevel enum for choosing the logging level NEW: (#6309) Introduce DReturnType/DOKReturnType/DErrorReturnType for DIRAC return structures CHANGE: (#6299) Set OpenSSL ciphers to "DEFAULT:@SECLEVEL=1" NEW: (#6299) Experimental support for Python 3.10 NEW: (#6299) Suppress InsecureRequestWarning when using dirac-configure with HTTPS servers NEW: (#6299) Support overriding ciphers and methods with environment variables when using HTTPS CHANGE: (#6298) Replace suds-jurko with suds CHANGE: (#6297) Use psutil.net_if_addrs for Network.discoverInterfaces NEW: (#6291) adding general flag for disabling Monitoring for Agents (used by JobAgent) CHANGE: (#6286) Default proxy length is 2048 bits CHANGE: (#6286) remove RFC argument for proxy generation *WorkloadManagement FIX: (#6338) Log error if CE Platform is missing. CHANGE: (#6336) Remove CPU(MHz) from CacheSize(kB) job parameters *Resources FIX: (#6320) Forward jobDesc to innerCE in PoolCE FIX: (#6318) Set HOME in SingularityCE FIX: (#6317) Return proper error if provider value missing in CloudCE CHANGE: (#6283) GFA2_SRM2 does not use anymore StorageElements/DefaultProtocols option in the CS (relies on specific OutputProtocols config) CHANGE: (#6283) explicit distinction between PluginName and ProtocolSection name in all the StorageElement components CHANGE: (#6283) API change for the StorageElement. See https://github.com/DIRACGrid/DIRAC/pull/6283 for details CHANGE: (#6283) StorageElement favors local plugins *WorkloadManagementSystem FIX: (#6288) ElasticJobParametersDB: do not return parameters that are not needed *tests FIX: (#6287) replication transformation test: avoid target==source FEAT: (#6287) transformation tests: allow -disk SEs, avoid RAL, RALPP, CESNET (don't work for me, YMMV) *MonitoringSystem FIX: (#6286) ElasticSearchDB generates full index names based on UTC FIX: (#6286) DataOperationSender works on a copy of the documents CHANGE: (#6286) DataOperationSender fails if the master accounting system fails [v8.0.0a27] *ConfigurationSystem CHANGE: (#6285) Make GOCDB2CSAgent functionMap less private *Core CHANGE: (#6281) Removed Core.Utilities.MemStat module (use psutil module instead) CHANGE: (#6281) Removed possibility to Register a DISET service with a secondary name *Resources FIX: (#6275) Conditional FC ProxyPlugin checks the ThreadConfig *DataManagementSystem FIX: (#6268) Fix FC change{group,owner,mode} for multiple LFNs *WorkloadManagement NEW: (#6267) ContainerBind option can be set on the per-site and per-CE level *WorkloadManagementSystem FIX: (#6256) Making getJobParameters handle specific parameters CHANGE: (#6178) removing filtering user sites by platform during the CHECKING state *FrameworkSystem NEW: (#6248) Add StdoutJsonBackend FIX: (#6246) dirac_login HTML error hiding the actual error [v7.3.25] *Resources FIX: (#6275) Conditional FC ProxyPlugin checks the ThreadConfig *DataManagementSystem FIX: (#6268) Fix FC change{group,owner,mode} for multiple LFNs *WorkloadManagement NEW: (#6267) ContainerBind option can be set on the per-site and per-CE level *FrameworkSystem NEW: (#6248) Add StdoutJsonBackend FIX: (#6246) dirac_login HTML error hiding the actual error [v8.0.0a26] *Resources FIX: (#6242) remove REST information from the ARC6 pilot job references to bind them with jobs FIX: (#6222) dirac_resource_info - restored the SE lookup logic FIX: (#6218) ARCCE.getARCJob() becomes protected FIX: (#6215) ARC6 typo issues FIX: (#6215) ARC writeXRSL becomes protected *WorkloadManagement FIX: (#6241) Fix Platform key error in SD when CheckPlatform=True *DataManagementSystem FIX: (#6240) DatasetManager: bug fix in getDatasetsInDirectory() method FIX: (#6209) Built-in metakeys were incorrectly modified in a MultiVO environment. This fix keeps them intact while modifying user-supplied metadata names by append a VO suffix, as before. *tests NEW: (#6239) added tests for the dirac-resource-info command *Interfaces CHANGE: (#6228) Make tmpdir for local cache instead of using CWD directly when replicating *Core FIX: (#6227) Increase VOMS command timeout FIX: (#6204) Fixing errors, adding tests for TimeUtilities module FIX: (#6128) Discovery of HTTPS methods from parent classes *MonitoringSystem FIX: (#6214) Fixing errors blocking commission of data operations to Elasticsearch *ResourceStatusSystem FIX: (#6201) convert ComputingElement to CE before searching in the configuration *WorkloadManagementSystem FIX: (#6196) Fixing errors, adding mapping for ESJobParameters CHANGE: (#6193) new method to submit pilots from ARC CHANGE: (#6161) removing the possibility to get the optimizers to run with a vo plugin [v8.0.0a25] *CI NEW: (#6189) max open files for dirac CI container to 8192 *Core CHANGE: (#6188) Propagate original proxy strength when generating new proxies *WorkloadManagementSystem CHANGE: (#6154) Addded timestamp to elasticjobparameters and changed parameter values into strings [v8.0.0a24] *Core FIX: (#6179) better report of error when importing Module FIX: (#6167) Replaced old vomses/vomsdir code to be able to handle cvmfs vomses. See https://github.com/DIRACGrid/DIRAC/issues/6150 FIX: (#6151) Replace deprecated threading.activeCount with active_count FIX: (#6149) use AnyThreadEventLoopPolicy asyncio policy FIX: (#6144) Removing time dependence of TimeUtilities Module NEW: (#6137) add https to issuer address automatically FIX: (#6132) fix error reporting in TornadoService CHANGE: (#6091) changed name and simplified repetitive methods of Utilities.TIme CHANGE: (#6084) BaseRequestHandler encode/decode static methods *MonitoringSystem FIX: (#6175) MonitoringReporter - late binding to MonitoringDB FIX: (#6076) Removing multiplication in timestamp when committing to monitoring *WorkloadManagementSystem FIX: (#6173) restore /LocalSite/InstancePath for the case of JobAgents (see issue #6143) FIX: (#6162) use the JobMonitoringClient to check for job parameters (that can be in the ES backend) NEW: (#6134) adding a test to show the modifications made to the JDL before being inserted in the JobDB *RequestManagementSystem NEW: (#6172) Add option to automatically cancel Scheduled requests after a defined time period. Default behaviour unchanged. Relates to #6136 CHANGE: (#6114) Replace BLOBs with TEXTs for ReqDB *docs NEW: (#6147) add info on installing non-released code from within the pilot *Resources/Cloud FIX: (#6146) userData need to be in "ascii" string format to meet the requirement of "create_node" when creating VMs from cloud resources *FrameworkSystem FIX: (#6138) dirac_login: avoid misleading error message, fixes issue #6096 FIX: (#6135) dirac_login: Pass proxy as string to writeToProxyFile() NEW: (#6130) add the possibility to add a prefix before the actual log message for a given subLogger [v8.0.0a23] *Tests FIX: (#6122) restart Tornado after updating the configuration *Core FIX: (#6120) don't truncate first character in cli doc strings FIX: (#6103) Race condition which causes multiple connectioons to be opened when the connection to an executor drops CHANGE: (#6103) Improve logging split between message/varmessage in execuctors FIX: (#6083) Grid: BDII make ldapsearch call forward compatible with newer ldapsearch versions *DataManagementSystem NEW: (#6115) Introduces an option 'ForceIndexedMetadata' to disable the creation of unindexed metadata. Default: False, i.e. preserves the current default where both indexed and unindexed metadata are allowed. Relates to #6029. *Resources FIX: (#6111) Error in the condition to get FlavorName FIX: (#6110) use identity instead of subject of proxy in CloudCE FIX: (#6102) remove decode from proxy in CloudCE FIX: (#6095) use ceType instead of ceName for CE parameters FIX: (#6088) autodetect cvmfs proxy in default cloudce template FIX: (#6083) ARCCE make ldapsearch call forward compatible with newer ldapsearch versions FIX: (#6080) Adapt StompMQConnector to Stomp 8 *tests FIX: (#6104) typo in testJobDefinitions for new Cloud test NEW: (#6081) added hackathon test for CloudComputingElement *WorkloadManagement FIX: (#6103) Avoid producing massive log output in OptimizationMind *Elasticsearch NEW: (#6094) option op_type in index API for Elasticsearch [v8.0.0a22] *Web FIX: (#6046) Fix downloading sandboxes from the Job Manager in the WebApp *Resources CHANGE: (#5996) release stomp version NEW: (#5996) Simple stomp interface [v8.0.0a21] *WorkloadManagement FIX: (#6066) In case of a pilot version not matching the production version, the JobAgent expects the error message to contain the words "Pilot version does not match", so the error message was updated accordingly. *WorkloadManagementSystem FIX: (#6065) Don't crash if systemCall fails *Resources NEW: (#6061) add SRM+HTTPs as a default TPC protocol in SRM FIX: (#6060) use ceType instead of ceName for CE parameters FIX: (#6052) string format for future in PoolCE *Core NEW: (#6017) TornadoBaseRequestHandler define a class logger with an extra attribute "tornadoComponent" NEW: (#6017) DIRACDB can now use a parentLogger instead of direct gLogger *All CHANGE: (#6017) services propagate their local loggers to the DB CHANGE: (#6017) tornado services use local logger [v8.0.0a20] *docs FIX: (#6041) Added (renewed) jobs and pilot State Machines drawings *DataManagementSystem NEW: (#6040) bitterly implement multihop NEW: (#6015) BringOnline setting for FTS transfers can be SE dependant *Resources FIX: (#6039) HTCondorCE: make sure proxy is available when running condor_rm NEW: (#6038) add CloudComputingElement *WorkloadManagementSystem CHANGE: (#6038) add hook for pilot cleanup *RequestManagementSystem FIX: (#6033) if the request ID is an integer the test `is digit()` throws an exception *Core FIX: (#6031) Allow rootPath to be overridden with DIRAC_ROOT_PATH environment variable *FrameworkSystem FIX: (#6007) ComponentSupervisionAgent: skip Tornado service *DMS/WMS/Monitoring FIX: (#6006) Fixed errors in Data Operation Monitoring and PilotsHistory from hackaton *MonitoringSystem CHANGE: (#6002) Replaced the flags for monitoring with a new unified flag system *Monitoring/Core CHANGE: (#5944) Divided ComponentMonitoring into AgentMonitoring and ServiceMonitoring [v8.0.0a19] *WorkloadManagementSystem FIX: (#6004) StalledJobAgent can force status from Submitting to Failed FIX: (#6004) ElasticJobParametersDB: make sure the docID is less than 512 characters *WMS/Monitoring NEW: (#5999) Added a PilotsHistory Monitoring *Subsystem FIX: (#5994) allow to run Watchdog on non x86 platforms *WorkloadManagement CHANGE: (#5993) dirac-wms-get-cpu-normalization depends on db12 multiple_dirac_benchmark *Core FIX: (#5985) Set default value of BaseRequestHandler.COMPONENT_NAME *DataManagementSystem FIX: (#5984) fix a race condition when running multiple FTS3Agents in parallel *RequestManagementSystem FIX: (#5960) moved REA initialization inside the initialize() method *MonitoringSystem FIX: (#5955) Fixed data operation and pilot submission monitoring from certif Hackaton [v8.0.0.a18] *Core FIX: (#5983) fix: allow for agents to loop every 10 seconds FIX: (#5980) Limit reoccurrences of subprocess unicode error *WorkloadManagementSystem FIX: (#5983) can't change the polling time of JobAgent from inside an already running agent FIX: (#5982) Stop fetching jobs when no more slot available in the inner PoolCE FIX: (#5982) Make sure dirac-jobexec does not use the server certificate to execute the workflow FIX: (#5973) update PoolCE.InnerCESubmissionType from JobAgent FIX: (#5950) escape time stamp in setHeartBeatData *MonitoringSystem NEW: (#5974) TornadoMonitoring handler *docs FIX: (#5969) added better docs for JobShare and priorities *FrameworkSystem CHANGE: (#5965) removed gMonitor FIX: (#5941) dirac-login uses the correct proxy to add VOMS attributes *RequestManagementSystem FIX: (#5958) dirac-rms-request takes into account PEP-515 FIX: (#5951) fix variable referenced before assignment in RequestTask FIX: (#5951) take into account that targetSEList can be empty in the operations *DataManagementSystem CHANGE: (#5954) DFC dump feature can dump more than one SE at the same time *Accounting/Monitoring FIX: (#5945) Fixed the plotter for pilot submission accounting and other small fixes [v8.0.0a17] *WorkloadManagementSystem FIX: (#5937) using a DErrno for no match found CHANGE: (#5933) removed last SubmitPool entries FIX: (#5907) Matcher: fixed parsing of (required) tags *Resources FIX: (#5935) File protocol listDirectory returns metadata CHANGE: (#5919) GFAL2_StorageBase redefine ECOMM if it does not exist *RequestManagementSystem FIX: (#5932) filter by SourceSE in ReplicateAndRegister preparation FIX: (#5932) Change RMS.Operation sourceSEList and targtSEList default to empty list FIX: (#5926) removing python2 support *FrameworkSystem CHANGE: (#5931) DBs stop using BLOB types in favor of TEXT types NEW: (#5901) move ProxyManager to HTTPs *DataManagementSystem NEW: (#5927) TornadoDataIntegrityHandler *Tornado NEW: (#5925) added TornadoWMSAdministratorHandler [v8.0.0a16] *tests CHANGE: (#5900) use dirac-login instead of dirac-proxy-init for tests *FrameworkSystem NEW: (#5899) dirac-login (AS) added VOMS extension NEW: (#5896) dirac-login try to connect to DIRAC AS if no local certificate *Core FIX: (#5898) print error details/callstack in BaseRequestHandler [v8.0.0a15] *docs CHANGE: (#5895) moved docs of ComputingElement from ConfReference to dirac.cfg CHANGE: (#5875) add more details about the PushJobAgent configuration FIX: (#5868) update DFC components description NEW: (#5867) Added general service configuration options description including Authorization options *WorkloadManagementSystem NEW: (#5894) Added Account option for SLURM CHANGE: (#5876) HeartBeatTime is set every time a status is changed by the job itself CHANGE: (#5840) Replace BLOBs with TEXTs for JobDB and PilotAgentsDB *Core CHANGE: (#5883) Show callstack in service logs when returning S_ERROR CHANGE: (#5872) Minimum Python version is now 3.9 FIX: (#5870) allow to use `deprecated` decorator with classes without predefined `__init__` function. *WorkloadManagement FIX: (#5875) add proxy in PushJobAgent and enhance it *Accounting CHANGE: (#5873) decrease some logs verbosity *All CHANGE: (#5870) use DIRAC.Core.Base.Script.Script instead of DIRAC.Core.Utilities.DIRACScript.DIRACScript CHANGE: (#5870) add deprecation message *FrameworkSystem CHANGE: (#5865) remove unused 'rfc' argument NEW: (#5862) add password mask to dirac-login *ResourceStatusSystem FIX: (#5864) return S_OK/S_ERROR in PublisherHandler.getSite *MonitoringSystem FIX: (#5860) Changed name of methods in PilotSubmissionMonitoringPlotter NEW: (#5807) Move DataOperation to Monitoring [v8.0.0a14] *DataManagementSystem FIX: (#5858) FileCatalogCLI - removed faulty evaluation of undefined option to the rebuild command. Fixes #5759 *Core FIX: (#5857) TorandoREST ignore arguments that not defined in the target method FIX: (#5857) dirac-info show "None found" instead of exception CHANGE: (#5854) Treat X509 data as str instead of bytes *FrameworkSystem FIX: (#5857) use payload keyword argument for handle_response method CHANGE: (#5857) use DIRAC CAs location to request DIRAC AS FIX: (#5857) fix message in dirac-login after authorization FIX: (#5854) UserProfileManager usage with JEncode *Resources CHANGE: (#5857) verify CAs when request IdP by default FIX: (#5856) Changed the nomenclature for PilotSubmissionMonitoring *WorkloadManagement FIX: (#5854) JobDB usage with JEncode *Production FIX: (#5854) ProductionManager use with JEncode *WorkloadManagementSystem FIX: (#5848) WMS DBs: do not try to decode when it's not bytes CHANGE: (#5846) added Modules and PipInstallOptions to SiteDirector/Pilot option *ConfigurationSystem FIX: (#5844) VOMS2CSAgent correctly takes into account multiple CAs for users with multiple DNs *MonitoringSystem FIX: (#5838) Fixing nomenclature and other issues with PilotMonitoringSubmission *Tornado FIX: (#5837) better logging formatting *TransformationSystem NEW: (#5829) Added TornadoTransformationManager service [v8.0.0a13] *FrameworkSystem CHANGE: (#5836) added a flag for disabling the use of SecurityLogging service *Core CHANGE: (#5833) JEncode bytes, tests FIX: (#5810) do not allow arbitrary code execution in JEncode *ConfigurationSystem CHANGE: (#5833) align with JEncode changes *ResourceStatusSystem NEW: (#5830) Added tornado version for the three RSS services *TransformationSystem CHANGE: (#5828) TransformationDB: removing BLOB in favor of TEXT for non-binary data FIX: (#5822) TransformationDB: removed old compatibility layer with mySAM tables FIX: (#5796) Convert fileID / taskID to fix issue when task status is updated in transformation treatment *docs FIX: (#5825) fix indentation in the doc related to the server installation CHANGE: (#5823) add further details about the server installation process with python3 *WorkloadManagement CHANGE: (#5824) enable logs from db12 in dirac-wms-cpu-normalization *Interface FIX: (#5818) send only single jobID to checkJobStateTransition Thank you for writing the text to appear in the release notes. It will show up exactly as it appears between the two bold lines Please follow the template: Thank you for writing the text to appear in the release notes. It will show up exactly as it appears between the two bold lines Please follow the template: *WorkloadManagementSystem FIX: (#5816) Allow transition Rescheduled->Failed *Subsystem NEW/CHANGE/FIX: (#5804) explanation For examples look into release.notes *MonitoringSystem CHANGE: (#5788) moving pilot submission to monitoring *Docs FIX: (#5711) updated/simplified the documentation for adding new components [v8.0.0a12] *WorkloadManagementSystem FIX: (#5793) better use the job state machine when setting the job status, in particular when it comes from failover requests that may be inserted between 2 successful updates. FIX: (#5751) allow Stalled, Running and Matched jobs to go to Reschedule FIX: (#5751) JobAgent forces the reschedule of jobs in case of exceptions FIX: (#5746) convert inputs from tuple to list in JobAgent before submitting to a CE NEW: (#5714) added WMSDashboard and diracLogsDashboard in NDJSON format. FIX: (#5714) removed old index patterns in WMSDashboard.json, updated documentation. *DataManagementSystem FIX: (#5791) Fix pickle error in dirac-dms-directory-sync *WorkloadManagement FIX: (#5789) JobSanity doesn't handle input sanboxes named "LFN:" *TransformationSystem Fix: Restore task state "Scheduled" *Resources CHANGE: (#5774) SingularityComputingElement: install python3 version NEW: (#5766) allow to overwrite input/output storage plugin parameter with the CS CHANGE: (#5766) add "xroot" to the list of input protocol for the XROOT plugin *Interfaces CHANGE: (#5772) DIRAC APIs check if a JobState transition is allowed *Subsystem FIX: (#5768) dirac_production_runjoblocal.py: moved to py3 Pilot install *All CHANGE: (#5764) use Client instead of RPCClient *HTTPS FIX: (#5762) Replace DIRAC.Core.DISET.TransferClient with DIRAC.Core.Tornado.Client.ClientSelector *FrameworkSystem FIX: (#5757) rename downloadablePersonalProxy value to allowProxyDownload FIX: (#5748) Get Tornado's port from the CS when adding URLs *docs CHANGE: (#5756) add document image source files *Core FIX: (#5743) removed isPy3VersionNumber function [v8.0.0a11] *WorkloadManagement NEW: (#5736) PushJobAgent to prepare jobs for sites with no external connectivity NEW: (#5736) RemoteRunner to execute applications remotely (submitting them to remote CE) *Resources CHANGE: (#5736) add inputs and outputs parameters in ARC.submitJob() CHANGE: (#5702) ParallelLibrary: Embed the executable in the srun wrapper *Workflow CHANGE: (#5736) integrate RemoteRunner *WorkloadManagementSystem FIX: (#5735) securityLogging: zip after 1 day CHANGE: (#5728) Replace DB12.py with the db12 package FIX: (#5725) ElasticJobParametersDB: get in output all entries CHANGE: (#5724) JobAgent and JobWrapper: only reporting changes on major status when needed *ResourceStatusSystem FIX: (#5734) CSHelpers: some sites have no resources *HTTPS FIX: (#5730) Add URLs to CS when installing HTTPS services *Core FIX: (#5726) TornadoBaseClientcorrectly checks CA FIX: (#5717) encodeDict when the dict keys have mixed types CHANGE: (#5698) Replacing Elasticsearch python client with OpenSearch. FIX: (#5694) do no look anymore for __RCSID__ FIX: (#5683) ThreadSafe.Synchronizer: Add target method docstring to preserve docstring of decorated function *ConfigurationSystem FIX: (#5701) fix an exception in the VOMS2CSAgent when a user joins a second VO FIX: (#5684) fix CSGlobals.Extensions class *TransformationSystem FIX: (#5690) fix more flaky tests for RequestTasks [v8.0.0a10] *ConfigurationSystem CHANGE: (#5682) Bdii2CSAgent: add InjectSingleCoreQueue option to automatically create single core equivalents for MutliCore Queues , kind of fixes #5582 CHANGE: (#5682) Bdii2CSAgent: CEs in the BannedCEs list are no longer updated by the agent, previously this list only concerned CEs that are not already in the Configuration, fixes #5224 CHANGE: (#5682) dirac-admin-add-resources: add --onecore option to automatically create single core equivalents for MutliCore Queues FIX: (#5682) dirac-admin-add-resources: fix the query about adding new CEs, this can now be answered in the negative *Resources FIX: (#5674) decode the http response in WLCGAccountingHTTPJson CHANGE: (#5643) HTCondorCE submits jobs with -spool option when a local schedd is used to spool the pilot wrappers that can be deleted afterwards *Core FIX: (#5671) DIRAC.isPy3VersionNumber always returns True FIX: (#5656) Tasks never get removed from the ExecutorState if an UnrecoverableTaskException is raised CHANGE: (#5629) use DIRAC_DEBUG_M2CRYPTO to enable SSL debugging in tornado *RMA FIX: (#5670) If a job was Killed, set it Killed at request completion *TransformationSystem FIX: (#5666) fix RequestTasks flaky tests NEW: (#5642) Allow MultiOperation body to fetch values from the task dict NEW: (#5642) Introduce Body Plugins for DMS transformations FIX: (#5630) TransformationCleaningAgent 'deletes', do not 'remove' jobs *WorkloadManagement FIX: (#5665) Python 3 support in SSHComputingElement FIX: (#5657) Bad error handling in OptimizationMindHandler.exec_taskError *Docs CHANGE: (#5663) removed a bunch of py2 related documentation *MonitoringSystem FIX: (#5658) Wrong link on the docs page for the WMSHistory dashboard. Fixed the link and edited some of the text. NEW: (#5634) JSON file for WMS monitoring dashboard *docs FIX: (#5654) fix trailing-whitespace *FrameworkSystem FIX: (#5652) SystemAdministratorCLI: get DIRAC version from Extensions CHANGE: (#5641) proxy expiration emails are sent from an address taken from the Service/Agent config instead of the DB NEW/CHANGE/FIX: (#5627) avoid duplication of code in TornadoBundleDeliveryHandler FIX: (#5619) continue on empty service list in getComponentsStatus *DataManagementSystem FIX: (#5651) dirac-dms-directory-sync: The script will now try all files and not end a thread if one of the files assigned to be uploaded or downloaded to a thread fails. The script will also print out all failed files, and their error at the end of a thread. A few log messages have been corrected as well. *ResourceStatusSystem FIX: (#5647) specifying the gocDB type for known types *WorkloadManagementSystem CHANGE: (#5643) Add functions in WMSUtilities, move content of getPilotLoggingInfo() and getGridJobOutput() in PilotManagerHandler *RequestManagementSystem NEW: (#5615) add SweepSize option to the ReqProxy [v8.0.0a9] *TransformationSystem FIX: (#5612) fix bug in TransformationCleaningAgent, that was NOT removing jobs but only deleting them *WorkloadManagementSystem FIX: (#5612) removing jobs instead of deleting them when bulk submission fails CHANGE: (#5570) better logging output in the JobStateUpdate *WorkloadManagement FIX: (#5610) Return error if JobPolicy.getControlledUsers returns an empty list FIX: (#5603) Don't run JobsStateMachine if newStat is '' FIX: (#5595) UnicodeDecodeErrors in JobDB.getJobParameters FIX: (#5585) Error logging when optimisation fails *FrameworkSystem FIX: (#5609) Cleaning old installations from a Python 3 based server FIX: (#5604) Issue installing DIRAC and extensions on servers running Python 3 FIX: (#5583) display correctly the help in sysadmin-cli CHANGE: (#5571) allow DIRAC AS return proxy by default *ConfigurationSystem FIX: (#5608) ServiceInterface.py - use ThreadPoolExecutor instead of ThreadPool to avoid thread leaks in the ConfigurationServer CHANGE: (#5596) do not add the master CS first in the list of CS when new slaves are added *Test FIX: (#5605) fix typo in the HTTPs Resources tests *Core FIX: (#5594) Extension ordering in DIRACScript *Resources FIX: (#5586) cloudinit-template - allow time in the monitor for the pilot to start FIX: (#5586) cloudinit-template - set DIRACSYSCONFIG to point to the pilot.cfg in order to allow its use in the user applications *docs CHANGE: (#5573) move the ConfReference DIRAC section part to dirac.cfg and add there OAuth 2 settings. FIX: (#5572) update nginx installation instruction NEW: (#5540) add documentation with information about new system APIs component Align `WebApp` installation manual with last changes (using BaseRequestHandler in WebAppDIRAC 5.0). The request Accounting -> Network -> Packet lost rate (magnified) causes an error: ``` ERROR: (#5569) Error while generating the plots Value keys 100 - IF(SUM(PacketLossRate)/SUM(entriesInBucket)*10 > 100, 100, SUM(PacketLossRate)/SUM(entriesInBucket)*10), 100 are not defined ``` This RP should provide correct pattern. *Accounting FIX: (#5569) provide correct pattern for reportMagnifiedPacketLossRate [v8.0.0a8] [v8.0.0a7] *WorkloadManagementSystem CHANGE: (#5566) JobParameters - generalize getting CS parameters for Cloud VMTypes also FIX: (#5563) PilotCStoJSONSynchronizer: do not look anymore in releases.cfg CHANGE: (#5555) SiteDirector submits only py3 pilots FIX: (#5541) old OptimizerModule can commit to ElasticJobParametersDB FIX: (#5529) QueueUtilities - do not stop queue instantiation in case of failures FIX: (#5518) CloudDirector - create only one pilot reference *Core FIX: (#5561) Fix overwriting files from an X509Chain object NEW: (#5543) Utility for parsing DIRAC version number to Py3 CHANGE: (#5537) Do not return failures when sending tasks to executors FIX: (#5530) wait that the CS is loaded before checking for master CS in tornado-start-all FIX: (#5516) Fix finding Dwatermark.png for the accounting watermark *Docs CHANGE: (#5559) Forces docutils to 0.17 because of bug with sphynx CHANGE: (#5546) do not calculate FQDN unnecessarily *FrameworkSystem NEW: (#5543) dirac-admin-update-instance converts version number to PEP-440 style if needed CHANGE: (#5537) ComponentInstaller: look also for HTTPs services FIX: (#5482) Allow Python 3 style pre-releases to be installed if the primary extension is a pre-release *ConfigurationSystem FIX: (#5535) VOMS2CSAgent: VOMS users that have multiple DNs and are suspended should no longer lead to "User not registered" errors and have their status correctly set *Resources FIX: (#5529) ComputingElement - remove unnecessary warning CHANGE: (#5528) Use PilotManagerClient to get CE status instead of interrogating PilotAgentsDB in HTCondorCE FIX: (#5518) OpenStackCE - removed buggy line NEW: (#5518) Utilities - added possibility to specify extra yum installable packages in the configuration NEW: (#5518) Utilities - added possibility to specify ssh connection to the VM in the configuration FIX: (#5518) cloudinit.template - use VMType as the Queue analog FIX: (#5518) dirac_resource_get_parameters - fixed bugs in the VMType getting logic *Interfaces FIX: (#5522) Dirac.py - JobMonitoring.getJobsSummary now returns a structure and not a string *RequestManagementSystem CHANGE: (#5521) getRequestFileStatus takes better into account cases of multiple operations on the same file CHANGE: (#5521) physicalRemoval checker expects an LFN and not a PFN *WorkloadManagement FIX: (#5516) Ensure valenc is bytes before decoding in retrieveOptimizerParam *TransformationSystem NEW: (#5514) Allow GroupSize parameter to be mutable NEW: (#5514) Add setGroupSize to command transformation CLI *SecurityLog CHANGE: (#5513) fed up with unreadable job logs... [v8.0.0a1] *FrameworkSystem FIX: (#5410) ProxyManagerClient - upload proxy without embedded DIRAC group *DataManagementSystem CHANGE: (#5391) The LcgFileCatalogClient has been removed Discard https://github.com/DIRACGrid/DIRAC/commit/dfbb53232dba40aff4bf0f04802877c51d078ab1 that resolve error described in previous hackathon https://trello.com/c/y7TgEuF7/40-multi-vo-tests. Now after the last lot of changes, this key returns correctly, so I canceled the last adjustment. *Interfaces FIX: (#5388) Discard https://github.com/DIRACGrid/DIRAC/commit/dfbb53232dba40aff4bf0f04802877c51d078ab1. [v8r0-pre4] NEW: Basic support for the OAuth/OIDC tokens *DataManagement CHANGE: (#5391) The LcgFileCatalogClient has been removed [v7r3p4] FIX: fixes from v7r2p28 *WorkloadManagement CHANGE: (#5429) Use python3 in VMDIRAC instances [v7r3p3] *Resources NEW: (#5396) Add Emies endpoint to the ARC CE *docs FIX: (#5424) some minor fixes for user documentation [v7r3p2] *WMS CHANGE: (#5404) use raw.githubusercontent URL for github raw files requests *TS CHANGE: (#5395) RequestTaskAgent uses objectLoader for RequestTasks CHANGE: (#5395) split RequestTasks and WorkflowTasks into distinct module *docs NEW: (#5218) Updated Python2 and Python3 server installation instructions CHANGE: (#5228) Documenting /WebApp/StaticResourceLinkDir option CHANGE: (#5404) use raw.githubusercontent URL for github raw files requests FIX: (#5404) use working git links for scripts [v7r3p1] FIX: Tag to push the PyPi deployment [v7r3] NEW: (#5000) Initial support for Python 3 server installations NEW: VMDIRAC separate project is merged into the core DIRAC project CHANGE: (#5035) use registerArgument to register positional arguments for scripts CHANGE: (#4715) removing env variable DIRAC_USE_NEWTHREADPOOL FIX: (#5035) use DIRACScript instead of Script FIX: (#5028) Replaced all the cases of BaseException use by Exception *Core NEW: (#5062) provide some docs and add registerArgument method to register arguments in Script NEW: (#4997) dirac-configure can now be ran without arguments when using Python 3 CHANGE: (#4937) removed dirac-agent, dirac-service, dirac-executor scripts (use '_' counterparts instead) CHANGE: (#5110) Removed Core.Utilities.Grid.getBdiiCEInfo function use Utilities.Glue2.getGlue2CEInfo. Also dropped ldapSite, ldapCluster, ldapCE, ldapCEState, ldapCEVOView, ldapService functions CHANGE: (#4903) Using former RSS State Machine as general State Machine CHANGE: (#5237) Use generic ObjectLoader.loadObjects() function in the specific Plotting.ObjectLoader class CHANGE: (#5244) dirac-configure: no upload attempt upon initial proxy generation CHANGE: (#5246) Drop MySQL._to_string() method CHANGE: Release constraint on SQLALchemy versions FIX: (#5210) show extremely small values in pie plots FIX: (#4997) Handle SIGINT correctly when reading certificate passwords FIX: (#5269) Use isinstance when checking types in MessageFactory FIX: (#5272) encode stub before base64 FIX: (#5326) fix bug in getSystemURLs: getOptionsFromCFG can return None CHANGE: (#5337) removed src/DIRAC/Core/scripts/dirac-install.py CHANGE: (#5372) RCSID is no longer access for Python 3 based installations *Configuration NEW: (#5062) add registerCmdArg to register arguments and group argument to group returned arguments NEW: (#5286) test new PathFinder methods NEW: (#5286) PathFinder - add checkServiceURL that check URL port and path NEW: (#5286) PathFinder - add getServiceURLs that return list type result NEW: (#5286) PathFinder - add getSystemURLs that return all services URLs for system CHANGE: (#5110) Dropped Glue2Only and Glue2URLS options from Bdii2CSAgent, Glue2 is now the only way CHANGE: (#5110) dirac-admin-add-resources: drop -g -G options, Glue2 is now the only way CHANGE: (#5110) Configuration.Client.Utilities functions getGridCEs, getSiteUpdates no longer take glue2 parameter CHANGE: (#5000) Deprecate CSGlobals.getInstalledExtensions, DIRAC.Core.Utilities.Extensions.extensionsByPriority should be used instead CHANGE: (#5000) Deprecate getCSExtensions, DIRAC.Core.Utilities.Extensions.extensionsByPriority should be used instead CHANGE: (#5286) PathFinder - getSystemSection do not use serviceTuple, add docs, use system and service instead of componentTuple FIX: (#5286) PathFinder - getGatewayURLs pass list to randomize when serviceName is empty FIX: (#5298) fix getServiceFailoverURL for multi url case FIX: (#5375) Handle S_ERROR result in CSAPI, docs *DMS CHANGE: (#5206) prints why dirac-dms-add-files failed *Framework: CHANGE: (#4303) Removed completely the SystemLogging CHANGE: (#5164) Remove HashTag table from Framework/UserProfile CHANGE: (#5164) Remove dataTypeRE argument when retrieving values from Framework/UserProfileClient FIX: (#5340) Fix using the System Administrator to install Python 3 releases of DIRAC from Python 2 *Interfaces CHANGE: (#5110) DiracAdmin: dropped getBDII* functions, which were looking for Glue1 information CHANGE: (#5110) removed dirac-admin-bdii-info command *Resources CHANGE: (#5314) Remove GlobusComputingElement *RMS CHANGE: (#5291) ReqDB don't cast datetime to strings *RSS NEW: (#5042) Multi-VO mode of operation support FIX: (#5335) VO parameter is optional. When omitted commands will act on all VOs. *WMS NEW: (#4903) Added WMS (Jobs) State Machine NEW: (#5214) Added a HTTPs JobManager service NEW: (#5214) Added a HTTPs JobMonitor service NEW: (#5214) Added a HTTPs JobStateUpdate service NEW: (#5314) Add a PilotStatus module to clearly define the pilot status in the code CHANGE: (#5214) Removed need for ThreadScheduler in JobManagerHandler CHANGE: (#5289) SiteDirector: do not send dirac-install if it's py3 pilot CHANGE: (#4884) JobDB: compress JDLs by default (no need for flag anymore) CHANGE: (#4937) removed StatesMonitoringAgent (use StatesAccountingAgent agent instead) CHANGE: (#5141) add an Aborted_HOUR column to the pilot efficiency table needed by the WebApp CHANGE: (#5178) JobMonitoring does not need to look into TaskQueue CHANGE: (#5246) remove pilots before start Test_PilotsClient test CHANGE: (#5318) Move SiteDirector.getQueues in QueueUtilities module FIX: (#5246) mock _escapeString in Test_Agent_PilotStatusAgent to pass test FIX: (#5246) escape values in PilotAgentsDB FIX: (#5246) decode values using decode() method in JobDB NEW: (#5314) Add a PilotStatus module to clearly define the pilot status in the code FIX: (#5325) Fix-up minor errors introduced during VMDIRAC merge NEW: (#5325) Tests for VirtualMachine DB & Service FIX: (#5342) Allow Platform parameter to be missing from the queue configuration *TS NEW: (#5327) Added TransformationFilesStatus module *tests NEW: (#5062) add Test_LocalConfiguration CHANGE: (#5046) don't use mail in the self generated certificates CHANGE: (#5301) added RALPP as a multi-VO test site and added env dump to job.log to ease debugging NEW: (#5321) added integration test for AccountingDB CHANGE: (#5362) Add proxy correct setup within wms-script.sh *docs CHANGE: (#5313) replace the theme by the sphinx_rtd_theme [v7r2p28] *Core FIX: (#5416) unless explicitly set, do not skip the CA checks *WorkloadManagement FIX: (#5431) Ensure invalid tasks don't get rescheduled by ExecutorDispatcher NEW: (#5430) add support for execution of pilot wrappers when the python executable is not set *RMS FIX: (#5432) ReqClient uses correct credentials to talk to ReqProxies FIX: (#5432) ReqClient uses proper WMS client insteand of RPCClient *Resources CHANGE: (#5428) asctime now includes microsecond precision when using the MessageQueueBackend for logging *DMS FIX: (#5427) adapt csv use to py3 *tests FIX: (#5427) adapt csv use to py3 [v7r2p27] *TS CHANGE: (#5413) TransformationDB: moved GroupSize from INTEGER to FLOAT *Resources FIX: (#5421) fix Slurm getJobStatus() method [v7r2p26] *Core FIX: (#5390) Handle S_ERROR in Service._executeAction NEW: (#5145) Add returnValueOrRaise and convertToReturnValue to simplify writing code with exceptions CHANGE: (#5394) Requests are now dropped if the backlog is too large FIX: (#5411) RPCClient and TornadoClient support proxyChain *Framework FIX: (#5410) ProxyManagerClient - upload proxy without embedded DIRAC group *WMS FIX: (#5414) JobCleaningAgent uses OwnerDN to put the sandbox removal request *DMS CHANGE: (#5284) FTS DB change (see https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-v7r2#new-status-for-fts3-files-and-jobs) NEW: (#5284) enable FTS archive monitoring NEW: (#5284) allow to use FTS activity based on FTS3Plugin FIX: (#5408) making use of freesize in WLCGAccountingJson if exists, instead of calculating as Total - usedsize *RMS CHANGE: (#5284) ReplicateAndRegister prefers disk replicas *Resources NEW: (#5402) DIRAC_GFAL_GRIDFTP_ENABLE_IPV6 to control gridftp behavior *TS FIX: (#5405) Transformation.setBody correctly checks all the operation tuples *tests NEW: (#5403) force commit format [v7r2p25] FIX: (#5373) Fix un-returned use of S_ERROR and S_OK *tests CHANGE: (#5380) Update DIRACCAProxyProvider test to use pytest FIX: (#5381) Adapt Test_FilePlugin.py for https://bugs.python.org/issue43219 *WMS FIX: (#5380) Fix bad merge in JobMonitoringHandler.py when producing v7r2p24 [v7r2p24] FIX: fixes from v7r1p46 *Resources FIX: (#5267) return error for StompMQConnector [v7r2p23] *WMS CHANGE: (#5348) PilotWrapper: added file:/cvmfs/dirac.egi.eu/pilot/ as location CHANGE: (#5356) SiteDirector - continue queues initialization if one of them fails *Resources FIX: (#5356) executeBatch - fallback to urllib if six module is not available FIX: (#5356) Host.py - return Message in case of non-zero status in getCEStatus FIX: (#5356) SSHBatchComputingElement - added forgotten instantiation of the BatchSystem plugin [v7r2p22] *Workflow FIX: (#5346) UnicodeEncodeError when application log contains non-ascii characters *Resources FIX: (#5345) Pass --cfg /tmp/pilot.cfg when reusing host DIRAC installation in SingularityComputingElement [v7r2p21] *Core FIX: (#5334) Use selectors(2) instead of select.select to avoid issues with file descriptors > 1024, fixes #5232 FIX: (#5336) ElasticSearchDB: use ca certs file if requested FIX: (#5328) ElasticSearchDB: work on a copy of the documents in generateDocs() *WorkloadManagement FIX: (#5332) Fix unassigning sandboxes when removing jobs from the transformation system *Resources NEW: (#5329) Added PBSResourceUsage unit test [v7r2p20] *Resources NEW: (#5316) Support Python 3 pilots in SingularityCE CHANGE: (#5310) changes some log levels in StorageElement CHANGE: (#5310) Default to using CERNVM4 in SingularityCE FIX: (#5310) Propagate the DIRAC and DIRACOS variables when not re-installing DIRAC in SingularityCE FIX: (#5315) Parallel Library Srun: remove chmod of executableFile *WMS FIX: (#5323) fix python3 incompatibility in InputDataByProtocol [v7r2p19] *Configuration FIX: VOMS2CSSynchronizer: if a user is in QuarantineGroup, remove it from all other VOMS groups [v7r2p18] *Framework FIX: ComponentSupervisionAgent: consistently use the actual "Setup", and not the "instance" to look up options. Note: you might need to change your config from "Production" to "DIRAC-Production" depending on you config. FIX: fix ProxyManagerClient docs *Resources NEW: removal throttle for Echo CHANGE: private methods in GFAL2_StorageBase are now protected *Transformation FIX: Python 3 serialisation error when calling JobMonitoringClient.getJobsStatus in WorkflowTasks.getSubmittedFileStatus FIX: DataRecoveryAgent: fix exception when there is exactly 1 done or failed job for a transformation [v7r2p17] *Framework FIX: (#5288) fix use of --excludeHosts in dirac-admin-update-instance NEW: (#5284) Introduce Local SubLogger *WMS CHANGE: (#5284) attempt to use Local subLogger NEW: (#5279) Added option for rescheduling Stalled Jobs, for certain sites *Resources FIX: (#5281) SSHComputingElement - treat properly the case of the pilots submitted to SSHBatch CE FIX: (#5281) SSHBatchComputingElement - use special SSHBATCH execution context FIX: (#5281) Host.py - put the actual execution host name/IP into the SSH_NODE_HOST environment [v7r2p16] *Configuration FIX: (#5266) fix RefresherBase log *Resources FIX: (#5271) only evaluate number of GPUs if necessary *WMS FIX: (#5271) Limiter: no error if the section does not exist *TS NEW: (#5268) add currentStatus parameter to TransformationClient.setTransformationParameter [v7r2p15] *WorkloadManagement FIX: (#5270) SandboxStoreClient tarfile mode [v7r2p14] NEW: Patch release is broken, use v7r2p15 instead *Core FIX: (#5265) Avoid pinging the database every time MySQL.__escapeString is called NEW: (#5263) Support datetime.date objects in JEncode *Framework FIX: (#5254) ComponentSupervisionAgent - use socket.getfqdn() instead of gethostname() *WorkloadManagement FIX: (#5264) TypeError when running dirac-wms-pilot-job-info CHANGE: (#5248) When removing Deleted jobs, they could not be removed if they had still an attached request. With this change, if the request is in final state, it will be removed and the job as well. *Transformation NEW: (#5265) Add columns keyword argument to TransformationClient.getTransformations NEW: (#5265) Add index on (TransformationID,Status) in TransformationFiles table NEW: (#5265) Add index on (TransformationID,ExternalStatus) in TransformationTasks table NEW: (#5265) Add index on (TransformationFamily) in Transformations table *Resources NEW: (#5256) SLURM can set gpus-per-task job requirement NEW: (#5256) added getNumberOfGPUs method to JobParameters utility FIX: (#5242) HTCondorCE: corrected ClassAd for requesting MP slots [v7r2p13] *Core CHANGE: (#5233) protect client doc generation against new importlib_resources *WorkloadManagement FIX: (#5245) Bytes confusion when using SandboxStoreClient with Python 3 *Resources FIX: (#5245) Bytes confusion when using SingularityComputingElement with Python 3 [v7r2p12] CHANGE: (#5219) Service Handlers discover DBs in Extensions *Core FIX: (#5226) DIRAC_USE_TORNADO_IOLOOP is set in the runsv file rather than the python starting script (#5223 ) FIX: (#5234) GOC: taking hostname from URL, if present *WMS NEW: (#5212) SiteDirector: added flag for pilots to install python3 DIRAC clients FIX: (#5220) JobPath executor: fix bug for per-job list of optimizers FIX: (#5229) JobAgent - protect against trying to send a None operation in a request CHANGE: (#5229) CHANGE: JobCleaningAgent - fixed some printouts FIX: (#5229) dirac-admin-ban-se - make it work with option --Site *Resources FIX: (#5231) fix path of the executable file when using a parallel library within SSHCE [v7r2p11] *Core FIX: (#5197) Fix handling Subprocess calls which produce invalid output *Framework FIX: (#5195) Avoid TypeError in ColorGenerator FIX: (#5198) ComponentSupervisionAgent: Pick up PollingTime from a base agent config *Resources FIX: (#5199) remove CPUScalingFactor calls *WorkloadManagementSystem FIX: (#5199) remove CPUScalingFactor calls FIX: (#5200) Matcher: use custom exception for Pilot Version Error [v7r2p10] *Python 3 FIX: (#5175) Improve "git describe" command used by setuptools-scm to ignore v6e7p27 *Core NEW: (#5180) Add callStack keyword argument to S_ERROR to allow object to be created much faster FIX: (#5182) Optimise ClassAd.asJDL to remove quadratic runtime dependence on the length of the output *Framework FIX: (#5187) ComponentSupervisionAgent: add necessary SystemAdministrator Authorization Settings in the docstring *Configuration CHANGE: (#5180) No longer include a full CallStack in the S_ERROR object returned by ConfigurationClient.getOption NEW: (#5185) Add DIRAC_FEWER_CFG_LOCKS environment variable to significantly improve multithreading performance in CS heavy workloads *WMS FIX: (#5184) JobDB: Getting InputDataModule from Operations section FIX: (#5184) JobCleaningAgent: get jobIDs old and recent, and combine *Resources FIX: (#5179) Handle the case where inputs is None in SSHComputingElement.submitJobToHost() CHANGE: (#5189) PoolComputingElement use concurrent.futures.ProcessPoolExecutor CHANGE: (#5189) PoolComputingElement: added a shutdown (called by JobAgent) *tests CHANGE: (#5186) AgentOptionTests: re-work ConfigTemplate discovery, should be transparent for the users, but work better for extensions [v7r2p9] *FrameworkSystem NEW: (#5173) Add option to exclude list of hosts from dirac-admin-instace-update CHANGE: (#5169) Deprecate checkTypeRe and HashTags in UserProfileClient FIX: (#5169) Python 3 compatibility in UserProfileClient [v7r2p8] *Core FIX: (#5158) Workaround for OSError when copying vomses directory with too many extended attributes *Framework NEW: (#5155) ComponentSupervisionAgent to Monitor running components. See Agent Documentation for details. *WorkloadManagement NEW: (#5122) multi-node allocation support via the ParallelLibraries and srun [v7r2p7] *Core CHANGE: (#5142) Move MySQL.ConnectionPool class to the module level CHANGE: (#5142) Move RequestHandler.ConnectionError to the module level FIX: (#5133) The RPC Client class should now be threadsafe FIX: (#5133) Correctly add RPC attributes to client objects in Python 3 FIX: (#5130) Avoid race when importing modules which can cause classes to be redefined *Transformation FIX: (#5135) TransformationBody update is too long for parameters [v7r2p6] *Core FIX: (#5000) Display thousands separators in PrettyScalarFormatter again [v7r2p5] FIX: fixes from v7r0p56, v7r1p39 *WMS CHANGE: (#5102) JobCleaningAgent will first DELETE and only then REMOVE jobs [v7r2p4] *Core NEW: (#5094) DIRAC.Core.Utilities.Extensions module for interacting with extensions in a Python 3 compatible way *tests NEW: (#5094) ./integration_tests.py helper script for running the integration tests locally [v7r2p3] FIX: Fixes from v7r054 and v7r1p37 FIX: (#5088) Fix various Python 3.9 deprecations *Core FIX: (#5095) Avoid KeyError when calling parseCommandLine *Framework CHANGE: (#5092) Use String(64) for DIRACVersion column in InstalledComponentsDB.HostLogging CHANGE: (#5098) in the run files created by ComponentInstaller, execute directly the deployed scripts *WMS NEW: (#5073) JobManager service now exposes a "RemoveJob" RPC call NEW: (#5073) implementing bulk indexing for jobs parameters CHANGE: (#5073) JobCleaningAgent will not remove those jobs that, while marked "DELETED", have still Operations to do in RMS *RequestManagement FIX: (#5091) Ensure BLOB columns are always passed bytes data *Resources CHANGE: (#5050) StompMQConnector uses a single connection *tests NEW: (#5086) Integration tests can be re-ran locally without re-installing the entire setup [v7r2p2] FIX: Fixes from v7r053 and v7r1p36 [v7r2p1] NEW: (#5041) Add Python 3 install docs to README FIX: (#5065) Explicitly convert variables assigned to range() into lists *Core CHANGE: (#5052) Added IS NULL mySql condition *WMS FIX: (#5039) dirac_wms_cpu_normalization: fix for python3 client, and stop looking for MJF CHANGE: (#5070) Using ThreadPool in StalledJobAgent *DMS NEW: (#5057) Allows to retrieve the non recursive size of a directory from the DFC CHANGE: (#5057) Factorize TornadoFileCatalogHandler and FileCatalogHander [v7r2] FIX: python 2-3 compatibility fixes NEW: (#4209) Extends the MonitoringSystem to support RMS and DMS/Agents/RequestOperations. CHANGE: (#4586) Use __future__ imports to get Python 3 like behaviour for imports, division and printing CHANGE: (#4796, #4797) removed obsoleted codes CHANGE: (#4823) Service handlers use class variables instead of global NEW: Moved to a src/ repository layout NEW: (#4906) Deploy releases to PyPI FIX: (#4921) use six.moves for urllib CHANGE: (#4958) use __doc__ for script help message, remove filecfg as argument from usage description FIX: (#5015) Add prompt-toolkit to setuptools metadata *Core FIX: (#4283) Added protection against empty certificates directory in dirac-install CHANGE: (#4582) Use concurrent.future.ThreadPoolExecutor by default. FIX: (#4606) Handle expected HTTP status codes in dirac-install (#4562) CHANGE: (#4630) Create symlinks to bashrc when using versions directory structure with dirac-install.py. Fixes #4565 NEW: (#4497) add histogram support to the Graph package. NEW: (#4497) adapt Graph package to python3 CHANGE: (#4581) Replace DIRAC.Core.Utilities.CFG with independent diraccfg module CHANGE: (#4671) Add an example for DIRAC_NO_CFG to bashrc. FIX: (#4736) dirac-install: use system tar command for extracting tarballs, fixes #4246 NEW: (#4677) X509Chain returns DN entry NEW: (#4677) introduction of Tornado based services NEW: (#4732) Support installing Python 3 clients using by passing --pythonVersion=3 to ./dirac-install.py CHANGE: (#4791) dirac-install with diracos: remove TERMINFO, RRD_DEFAULT_FONT, GFAL and ARC paths from bashrc FIX: (#4791) Late import of MonitoringReport NEW: (#4823) flipped default value of DIRAC_USE_JSON_DECODE env variable FIX: (#4843) X509Chain: fix downloading X509 certificates FIX: (#4870) ElasticSearchDB: fix exception handling when checking existance of indices FIX: (#4887) fix exception in Services when trying to use uninitialized gMonitor client while targeting ES monitoring NEW: (#4898) added a base DB class for all DB types NEW: (#4865) Fully support installation with setuptools CHANGE: (#4932) Rename DIRAC_VOMSES to X509_VOMSES in dirac-install.py CHANGE: (#4958) when import Script, a help message is formed from __doc__, not need to use Script.setUsageMessage(__doc__) FIX: (#4956) use logging.warning in favor of deprecated logging.warn CHANGE: (#4997) Use prompt-toolkit for password input when generating proxies CHANGE: (#4994) dirac-install.py should now be taken from management CHANGE: (#5010) Use ConfigurationClient in CSCLI and CSShellCLI CHANGE: (#5026) Remove DIRAC.Core.Utilities.ExitCallback *Configuration CHANGE: (#4866) split standard and tornado refresher in two files to avoid tornado dependency on the client CHANGE: (#4958) setUsageMessage in LocalConfiguration will parse received str value for building help message FIX: (#5008) HTTPS: allow to run master CS in parallel of other tornado services FIX: (#5016) harmonize https and dips version of ServiceHandler, and make doc consistent with reality *Framework CHANGE: (#4787) expose deleteProxyBundle call to ProxyManagerClient FIX: (#4850) MQ logging sends also the message,asctime and levelname field FIX: fix link to DIRAC install instruction and comments for WebApp FIX: (#4977) SystemAdministratorClientCLI - fix log message about responding hosts in sysadmin cli CHANGE: (#4956) renames of column names to avoid reserved keywords *Interfaces CHANGE: (#4799) remove runLocalAgent method from DIRAC API *Accounting NEW: (#4537) Add histogram of CPU usage to Job accounting NEW: (#4537) Implement the method used to retrieve data for creating histograms *WMS NEW: (#4367) Added option "CompressJDLs" to JobDB, which compresses JDLs before they are added to the JobJDLs table CHANGE: (#4428) removed the watchdog for mac, simplified linux CHANGE: (#4577) Removed the Pilot2 NEW: (#4799) JobWrapperTemplate kills the JobWrapper in case of Exception during Execution phase CHANGE: (#4799) Remove MultiProcessorSiteDirector (use standard SiteDirector) CHANGE: (#4799) stop setting JDL requirements MaxCPUTime, SubmitPools, GridRequiredCEs, Origin, JobMode CHANGE: (#4823) using JobReport and delayed send for JobStateUpdate FIX: (#4829) PilotCStoJSONSynchronizer - upload method uses WebDav NEW: (#4874) new agent PilotsSyncAgent for syncing pilot files. NEW: (#4874) new script dirac-admin-sync-pilots for syncing pilot files. CHANGE: (#4874) largely simplified PilotCStoJSONSynchronizer CHANGE: (#4874) don't call PilotCStoJSONSynchronizer from the CS NEW: (#4905) moving towards a single agent for WMS history CHANGE: (#4925) SiteDirector - removed using of SubmitPools requirement by default CHANGE: (#4925) SiteDirector, JobAgent - no more using ProxyManager tokens while user proxy retrieval NEW: (#4950) JobAgent adds GridCE and queue to Job Parameters CHANGE: (#4966) added flag to exclude the master CS from the list of servers synchronized by PilotCS2JSON FIX: (#5017) Fix uploading jobDescription.xml in upload sandboxes with Python 3 *DataManagement CHANGE: (#5019) Remove IRODSStorageElementHandler *Monitoring FIX: (#4768) Adapting RMSMonitoring type to ES7 FIX: (#4844) Correct the rounding of integers when computing buckets in MonitoringDB NEW: (#4898) added possibility to configure the period and so the indexing strategy CHANGE: (#4976) removed DynamicMonitoring (replaced by ActivityMonitoring) *ResourceStatusSystem CHANGE: (#4235) Removes locals() way of passing parameters NEW: (#4900) Remove old entries from history CHANGE: (#5002) RSS handlers: removed global objects *RMS CHANGE: (#4887) use dedicated ES monitoring flag RMSMonitoring *docs FIX: (#4740) make dirac-docs-get-release-notes.py python3 compatible CHANGE: (#4823) added some clarifications on starting the pilots CHANGE: (#4829) updated configuration in terms of security, added description of webdav FIX: (#4905) added config and instructions for Monitoring DBs (ES) configuration FIX: (#4973) fix link to DIRAC install instruction and comments for WebApp *tests NEW: (#4179) Set up Gitlab CI pipeline using Docker containers FIX: (#4677) Ignore tornado services and test DBs NEW: (#4677) new integration tests for Tornado based services NEW: (#4910) --runslow option on unit tests to allow faster local tests NEW: (#4938) added a helloworld test for the (yet to be implemented) cloud testing in certification CHANGE: (#4968) Change the defaults for tests (to MySQL 8 and ES 7) [v7r1p47] *Accounting FIX: (#5412) do not attempt to write buckets if there's nothing to write NEW: (#5392) Add AccountingMonitor permission FIX: (#5257) GROUP BY statement syntax to be compatible with more recent SQL reference (e.g. as used in MariaDB 10.5). This fix is backward compatible. *WMS FIX: (#5382) Fix the job matching algorithm used by the dirac-wms-match command *RequestManagement FIX: (#5260) Fix the GROUP BY syntax in the RequestSummaryWeb to be compliant with MariaDB 10.5. The fix is backward compatible with MySQL 5.7 [v7r1p46] *WorkloadManagement FIX: (#5364) convert Owner attribute into OwnerDN in PilotManagerHandler.getPilotStatistics FIX: (#5368) JobMonitoring - fix "Unknown column 'PilotJobReference'" error [v7r1p45] Data Management FIX: (#5290) Type error when calling FTS3ManagerHandler.getActiveJobs [v7r1p44] *WorkloadManagement CHANGE: (#5205) Make error message from SandboxStoreClient clearer when the sandbox is not found *DataManagement CHANGE: (#5211) FileCatalogClientCLI register File now takes a fixed number of arguments (6) including Checksum *Resources NEW: GFAL2_XROOTStorage - set XrdSecGSIDELEGPROXY=1 to enable third party copy *Interfaces NEW: (#5201) Implemented a function that allows programmatic listing of directories in the DFC. Meant to replace the never fully deployed listCatalog function. [v7r1p43] CHANGE: new features and fixes from v7r0p57 *WorkloadManagement FIX: (#5177) Look for dirac-jobexec instead of $DIRACROOT to decide when to copy the pilot.cfg to improve compatibility with v7r2 [v7r1p42] *Framework FIX: (#5168) Avoid minor caching bug which causes constraints to be violated when rapidly writing to UserProfileDB [v7r1p41] *Accounting FIX: (#5152) Bug in PilotSubmission efficiency plot *Framework NEW: (#5159) Add UserProfileClient.listStatesForWeb endpoint to make the Public State Manager faster CHANGE: (#5153) change SQLITE connection option for speed improvement *DataManagement FIX: (#5163) DatasetManager - Fix the syntax of the query to get dataset files FIX: (#5162) Improve listing error if file info is missing from DB *WorkloadManagement CHANGE: (#5161) SiteDirector - Combine CE and Queue name in PilotSubmissionAccounting to avoid duplicated queue name [v7r1p40] *Core CHANGE: (#5123) Read X509_CERT_DIR from central place *WorkloadManagement CHANGE: (#5143) Clarify that "tarfile failed with message" can be normal FIX: (#5146) SiteDirector._getPilotsWeMayWantToSubmit: fix return type in case of error *DataManagement CHANGE: (#5144) FileManagerPS has bulk method for getReplicas *Resources CHANGE: (#5137) HTCondorCE submits jobs with -spool option when a local schedd is used to spool the pilot wrappers that can be deleted afterwards [v7r1p39] *WMS CHANGE: (#5121) for HTCondor, the SiteDirectory write the executable in the globally defined working directory [v7r1p38] FIX: fixes from v7r0p55 *WMS FIX: (#5108) delete the local output sandbox tarfile in case it's still there FIX: (#5106) SiteDirector - use | operand for the union of tests CHANGE: (#5104) JobDB.setJobAttributes: use _update instead of _transaction (as not needed) [v7r1p37] *DataManagement NEW: (#5096) FTS3 plugins to alter TPC and source SE preferences [v7r1p36] *DMS FIX: (#5080) SQLAlchemy 1.4 support for FTS3DB FIX: (#5054) Submit FTS3 jobs with lowercase checksum *RMS FIX: (#5080) SQLAlchemy 1.4 support for ReqDB *Resources FIX: (#5077) Host.py - Fix for the case where no subprocess32 module found [v7r1p35] FIX: Fixes from v7r0p52 *Core NEW: (#5047) DIRAC_M2CRYPTO_DEBUG env variable for SSL debugging *WorkloadManagement FIX: (#5062) Move the CPU work left computation at the beginning of the JobAgent execution CHANGE: (#5053) Remove margins from the TimeLeft utility CHANGE: (#5051) if the HeartBeatTime of a job is not set, it cannot be seen as Stalled. Hence set the HeartBeatTime (if not set) when setting the StartExecTime CHANGE (#5051) (StalledJobAgent): don't try and get pilot status if "Unknown" *Interfaces CHANGE: (#5051) print the jobID in dirac-wms-job-logging-info *DMS CHANGE: (#5051) lower logging level for individual LFNs (from INFO to VERBOSE) to decrease amount of logged information. Replace with a summary [v7r1p34] FIX: Fixes from v7r0p51 patch release [v7r1p33] *DMS FIX: (#4960) FTS3 explicitly use verify_checksum NEW: (#4960) add --FTSOnly and --ExcludeSE options to dirac-dms-protocol-matrix *RMS CHANGE: (#4960) ReqClient prints the FTS3 job url directly [v7r1p32] *Core FIX: (#5021) Glue2: fix error when too many execution environments were looked for FIX: (#5021) Subprocess: when the systemCall failed, S_ERROR now contains a string in 'Message' instead of an exception object [v7r1p31] *Resources FIX: (#4996) Getting access to $_CONDOR_JOB_AD in HTCondorResourceUsage *WMS CHANGE: (#5001) JobAgent: if we don't match a job, independently from the reason, we wait a bit longer before trying again CHANGE: (#5001) JobAgent: moved check of availability of CE before doing anything else NEW: (#4998) JobWrapper - find executables from PATH if not using an absolute filename FIX: (#5009) As the job status is set AFTER the request is uploaded, it may happen that the request is executed BEFORE the status is set... Add thus a delay of 2 minutes for execution [v7r1p30] *Configuration FIX: (#4879) The loop in the siteCE mapping was exiting as soon as a mis-configured site was found. Fixed to continue the loop, while adding an error message. FIX: (#4970) Utilities - fixed bug in addToChangeSet(): use tuple instead 4 arguments FIX: (#4974) Utilities - fix setting of LocalCEType *WorkloadManagementSystem FIX: (#4972) fix the cpu work left computation in JobAgent when batch system information cannot be processed CHANGE: (#4978) Improved logging of the Matcher: adding pilotReference *Resources FIX: (#4955) SingularityComputingElement - better error message when the Singularity CE not found CHANGE: (#4972) add a condition in TimeLeft module to know whether the batch system relies on wallclock and/or cpu time NEW: (#4972) HTCondorResourceUsage module to get the time left in CERN-specific HTCondor resources. CHANGE: (#4935) HTCondorCE: add a periodic remove in case we are not using the local schedd [v7r1p29] FIX: Fixes from v7r0p49 patch release *Core CHANGE: (#4954) using argparse for a few scripts instead of getopt NEW: (#4957) ObjectLoader: do not fail if one module fails importing in recursive loading *Framework CHANGE: (#4964) dirac-proxy-init always uploads the proxy unless --no-upload is specified FIX: (#4964) ProxyManagerClient checks for group and groupless proxies in its cache *WMS FIX: (#4954) PilotCStoJSONSynchronizer - bug fixed in getting CE information CHANGE: (#4956) JobAgent - resorting to calculate time left with info in possession *Workflow FIX: (#4953) ModuleBase - remove deprecated function that does not do anything [v7r1p28] CHANGE: Revert #4919 - absolute imports in several modules *Configuration FIX: (#4945) CS.Client.Utilities: change exception caught to IndexError *WMS FIX: (#4947) SiteDirector: fix for case of empty tag *DMS CHANGE: (#4891) split staging and access protocols NEW: (#4891) FTS3 can submit transfers with staging protocols different from transfer protocols [v7r1p27] FIX: Fixes from v7r0p47 FIX: (#4919) Added from future import absolute_import when having conflicts when importing standard modules. Fixes issue #2207 *Core FIX: (#4918) Glue2: fix problem for CEs with multiple Queues, only one queue was discovered FIX: (#4926) ElasticSearchDB: fix exception handling when checking existence of indices *Configuration CHANGE: (#4926) Adding Pool as LocalCEType for MultiProcessor queues (invoked by BDII2CSAgent) *Framework CHANGE: (#4928) initialized, processMessage and flushAllMessages functions of Logging are deprecated *WMS FIX: (#4923) The StalledJobAgent's behaviour changed (for bad) as only jobs stalled for more than 24.5 hours were checked before being set Failed... Restore v7r0 behaviour with immediate check CHANGE: (#4926) JobAgent: getting CEType only from LocalSite/LocalCE config CHANGE: (#4926) JobAgent: flipped the default for fillingMode CHANGE: (#4926) SiteDirector: The max number of jobs in filling mode is not set anymore by the SD. The option MaxJobsInFillMode was removed *Resources FIX: (#4931) InProcess/SingularityComputingElement - Fix calling _monitorProxy as a periodic task *Monitoring NEW: (#4924) added possibility to login to ES via certificates. *docs NEW: (#4853) user management, ProxyManager CHANGE: (#4853) move Registry from ConfReference to dirac.cfg CHANGE: (#4853) move CommandReference content to scripts docs [v7r1p26] FIX: Fixes from v7r0p46 *Resources FIX: (#4909) use grid CA to download the srr file FIX: (#4904) ARCComputingElement: fixed a few places where unicode literals have leaked: wrapped in str *WMS FIX: (#4904) JobAgent: using internal function for rescheduling jobs [v7r1p25] FIX: Fixes from v7r0p45 *Core CHANGE: (#4890) logging failing MySQL commands *WMS FIX: (#4889) fix for retrieving the pilot reference when it's an ES JobParameter FIX: (#4889) fix for using the ES JobParameters for static data of heartbeats (app std out) FIX: (#4897) added timestamp to pilot.json [v7r1p24] FIX: fixes from v7r0p44 *TransformationSystem FIX: (#4883) fix to allow to pass a list of sourceSE to dirac-transformation-replication [v7r1p23] FIX: Fixes from v7r0p43 *Configuration FIX: (#4869) Bdii2CSAgent: remove constant update of NumberOfProcessors parameter. Comparison between int and str lead to "new" value every run. *WMS FIX: (#4868) JobLoggingDB: fix for backward compatibility *Monitoring CHANGE: (#4872) better, faster aggregations. Always using term queries for keyword fields, and other less important changes CHANGE: (#4872) do not return error if trying to delete non-existing index *tests NEW: (#4872) added integration test for MonitoringDB [v7r1p22] *WMS NEW: (#4852) new module JobMinorStatus.py to put some standard minor statuses CHANGE: (#4852) JobStateUpdateHandler - set startTime of job (in attributes) as from when the (#4852) job is Running (was when the application started). As agreed in a mail discussion... CHANGE: (#4852) JobWrapper - send failover requests and accounting information after setting the final status [v7r1p21] *ConfigurationSystem FIX: (#4854) Fix typo in CERNLDAPSyncPlugin *Resources FIX: (#4848) Fix SingularityComputingElement for non-server-style installations [v7r1p20] *Core CHANGE: (#4842) TLS: read proxy flag from M2Crypto.X509 *Framework CHANGE: (#4832) NotificationHandler: sendMail: the avoidSpam parameter is deprecated. All emails (same subject, address, body) are now only sent once per 24h *WorkloadManagementSystem FIX: (#4839) There was a bug handling the bulk status updates that was not updating correctly the job status although the job logging information was correct). This PR fixes this bug. It was tested in certification as a hotfix CHANGE: (#4839) internally always call the setJobStatusBulk() method in order to be consistent CHANGE: (#4839) in JobReport, only send statuses that are not empty, and remove handling of ApplicationCounter that is never sent nor used anywhere CHANGE: (#4834) JobStateUpdateHandler: use class variables instead of global *Resources FIX: (#4841) Fix using SingularityComputingElement with the host's DIRAC installation *Interfaces FIX: (#4834) Job.py: use --cfg option for JobConfigArgs *RSS FIX: (#4834) Added one field for retrieving the corrected order of columns [v7r1p19] FIX: Fixes from v7r0p40 *Configuration FIX: (#4821) Bdii2CSAgent: Updating CE information wasn't working because the list of CEs wasn't picked up from the right place (Site/CE option, instead of Site/CES/) FIX: (#4821) Glue2: fix association of CEs to sites if the CE associated sitename in the BDII differs from the Name given in the CS FIX: (#4821) dirac-admin-add-resources: Fix bug preventing the use of GLUE2 mode CHANGE: (#4636) BDII2CSAgent: stop looking for SEs in BDII *Resources FIX: (#4827) SingularityCE: Create pilot.cfg as part of the inner DIRAC install. *RSS FIX: (#4830) added vo field to ResourceStatusClient calls [v7r1p18] *RSS FIX: (#4815) Fix a problem with vo being returned in a middle to a list in toList [v7r1p17] *Configuration NEW: (#4784) VOMS2CSAgent introduce SyncPlugins to add extra or validate user information information NEW: (#4809) Add CERNLDAPSyncPlugin for VOMS2CSAgent *Accounting CHANGE: (#4798) multiply wallclock per the number of processors used *WMS FIX: (#4816) JobStateUpdateHandler - Fix bug when using elasticJobParametersDB *Resources FIX: (#4792) Initialisation of arc.Endpoint in ARCComputingElement *RSS FIX: (#4790) use createClient decorator for RSS clients [v7r1p16] *Resources FIX: (#4786) FileCatalogClient: correctly returning error in getReplicas() call [v7r1p15] *Configuration FIX: (#4766) Bdii2CSAgent: Fix exception if selectedSites was set, but some unknown Sites were found by the agent CHANGE: Bdii2CSAgent: if selectedSites is configured, also remove all CEs from unknown sites from the results *Framework CHANGE: (#4766) NotificationHandler: sendMail: the avoidSpam parameter is deprecated. All emails (same subject, address, body) are now only sent once per 24h *Resources FIX: (#4777) Mount the host's DIRAC installation in the container when using SingularityCE without InstallDIRACInContainer *RSS CHANGE: (#4767) SummarizeLogAgent uses list instead of tuples *WorkloadManagement CHANGE: (#4767) JobMonitoring uses cls/self attributes instead of global variable CHANGE: (#4772) JobMonitoring.getJobPageSummaryWeb can provide jobIDs according to the pilotJobReferences [v7r1p14] *Core CHANGE: (#4752) Glue2: the number of ldap searches has been reduced to three making lookups much faster CHANGE: (#4752) Glue2: return also unknown sites not found in the information provider for given VO CHANGE: (#4752) Glue2: fill value for SI00 for queues as well. Value is fixed at 2500 and will overwrite any existing value. CHANGE: (#4752) Glue2: architecture will now always be lowercase. CHANGE: (#4752) Glue2: Read the GLUE2ComputingShareMaxSlotsPerJob value and fill the NumberOfProcessors entry for the respective queue. Ignores CREAM CEs. A Ceiling for this number can be set by /Resources/Computing/CEDefaults/MaxNumberOfProcessors, defaults to 8. Solves #3926 CHANGE: (#4747) Added support for ElasticSearch 7. CHANGE: (#4747) Find cas.pem and crls.pem if they are local, before generating them. *Configuration: CHANGE: (#4752) Bdii2CSAgent: change email subject to agent name CHANGE: (#4752) Bdii2CSAgent: Make GLUE2Only=True the default CHANGE: (#4752) Bdii2CSAgent: Email also information about CEs not found in the specified information, CEs can be ignored with BannedCEs option, solves #1034 CHANGE: (#4752) dirac-admin-add-resources: printout CEs that are not known at the given information provider CHANGE: (#4752) dirac-admin-add-resources: default to Glue2, deprecate -G flag. Add -g flag to use glue1 FIX: (#4752) dirac-admin-add-resources: Do not exit if no new CEs are found, still look for changes of existing CEs *Monitoring CHANGE: (#4747) Eliminate types from MonitoringDB for support for ElasticSearch 7 *WMS CHANGE: (#4747) renamed ElasticJobDB.py in ElasticJobParametersDB.py CHANGE: (#4747) Eliminate types from ElasticJobParametersDB.py for support for ElasticSearch 7 CHANGE: (#4747) Use ES query DSL for searches in ElasticJobParametersDB.py, also for deleting entries *Resources CHANGE: (#4755) remove backward compatibility for BaseSE defined in StorageElements section (issue #3516) CHANGE: (#4755) Clearer error message related to LFN convention (issue #2790) [v7r1p13] *WMS CHANGE: (#4750) JobMonitoring casts for JSON *Framework CHANGE: (#4750) ignore serialization error for JSON in the Monitoring *RMS CHANGE: (#4750) ignore serialization error in ForwardDISET tests FIX: (#4745, #4746) transformation_replication - fix integration test [v7r1p12] *Core: FIX: (#4724) correctly pass the argument to the new ThreadPool in MessageBroker FIX: (#4737) modify logging when CAs & CRLs sync *Framework FIX: (#4728) Fix hashing of local CA and CRL bundles FIX: (#4737) add check of availability of directories for sync CAs in BundleDeliveryClient *Production FIX: (#4739) Fixes for JSON serialization *WMS FIX: (#4739) Fixes for JSON serialization *Transformation FIX: (#4739) TransformationManagerHandler: fixes for JSON serialization *ResourceStatusSystem CHANGE: (#4720) add a VO column to all tables as apart of the primary key and a default value='all' *Resources NEW: (#4721) SingularityCE: singularityCE: added possibility to run without re-installing DIRAC inside the container CHANGE: (#4733) SingularityComputingElement: Enable userns option in singularity if possible. CHANGE: (#4733) SingularityComputingElement: Use CVMFS as singularity fallback location if userns is enabled. *docs: CHANGE: (#4721) improved doc for SingularityCE options *tests FIX: (#4724) fix rss-scripts syntax [v7r1p11] *WMS FIX: (#4712) PilotStatusAgent was crashing due to a bug *Test FIX: (#4712) fix for main RSS system test [v7r1p10] Fixes from v7r0p32 CHANGE: (#4690) ComponentInstaller uses --cfg option [v7r1p9] Fixes from v7r0p31 *Resources FIX: (#4704) ARCComputingElement: fix use of gLogger, which was replaced by self.log [v7r1p8] *WMS NEW: (#4699) add a retryUpload option in the JobWrapper *DMS NEW: (#4699) add a retryUpload option to FailoverTransfer CHANGE: (#4693) Retry file upload for any error condition, not just file catalogue failure. *Resources FIX: (#4669) multiple minor style fixes *tests FIX: (#4695) Tests are modified so the MultiVO File catalog is not a master catalog anymore. Addresses #4682 . [v7r1p7] *Core FIX: (#4679) dirac-install is getting the globalDefaults.cfg from CVMFS if available FIX: (#4679) dirac-install - loads the installation configuration from etc/dirac.cfg if it is present NEW: (#4679) dirac-install-extension - new command to add an extension to an existing installation CHANGE: (#4679) dirac-deploy-scripts - added --module option to inspect only specified modules *WMS CHANGE: (#4674) DownloadInputData no longer fails if getFileMetadata fails for the first SE, tries others if available *DMS FIX: (#4678) Fix problem where a FileCatalog instance was not aware of SEs added by a different instance. SEManager.getSEName now refreshes if an seID is not known. Fixes #4653 *docs FIX: (#4667) some more info on duplications *tests FIX: (#4681) Test_UserMetadata makes dynamic lfns using VO [v7r1p6] *Framework CHANGE: (#4643) BundleDeliveryClient: inheriting from Client *SMS FIX: (#4645) StorageManagementDB - drop the tables in the correct order to avoid foreign key errors *TS NEW: (#4641) TransformationCleaningAgent will (re)clean very old transformations that are still in the system *tests FIX: (#4645) dropDBs uses real array; FIX: (#4645) remove readonly variables [v7r1p5] *Framework FIX: (#4640) Install of ES DBs: only search in *DB.py files [v7r1p4] Includes fixes from v6r22p30, v7r0p26 *Framework CHANGE: (#4632) dirac-sysadmin - handle installation possible also for ES DBs *tests NEW: (#4633) added xmltodict module - makes working with XML feel like working with JSON. [v7r1p3] Includes fixes from v6r22p29, v7r0p25 *Framework FIX: (#4611) ProxyManager, dirac-admin-get-proxy - allow bool type for vomsAttribute parameter on service side. Fixes #4608 [v7r1p2] Changes from v7r0p24 patch [v7r1p1] Changes from v6r22p28 patch *Framework CHANGE: (#4572) removed group from proxy expiration notification [v7r1] NEW: Add environment.yml file for preparing an environment with conda CHANGE: (#4507) autopep8 for the entire code stack *Core NEW: initial support for mysql8 NEW: (#4236) DISET - pass the client DIRAC version as part of the request description structure NEW: (#4274) Allow installation from a local directory FIX: (#4289) Use subprocess32 if possible for significantly better performance FIX: (#4289) TypeLoader should not be a singleton as it has a cache CHANGE: (#4362) RPC parameters are cast to list/tuples when needed FIX: (#4410) M2Crypto transport cast socket.error to strings CHANGE: (#4410) (JSON) rpcStub uses list instead of tuples NEW: (#4410) add utilities for JSON Serialization (strToIntDict and ignoreEncodeWarning) CHANGE: (#4354) ThreadPool - replaced by ThreadPoolExecutor(native python) FIX: (#4439) MySQLDB _connect method NEW: (#4510) Removed dirac-distribution (use docker image) CHANGE: (#4491) Streamline of CS helpers for sites and computing elements NEW: (#4538) MixedEncode - allows for json serialization transition CHANGE: (#4461) use M2Crypto by default CHANGE: (#4565) Source bashrc in wrapper script *Configuration CHANGE: (#4157) add new helper methods for IdP/Proxy Providers, OAuthManager. Add methods that read DN options in dirac user section NEW: (#4241) Slave configuration servers are automatically updated as part of the new configuration changes commit NEW: (#4241) Configuration Server - added forceGlobalConfigurationUpdate() interface to force all registered service to update their configuration NEW: (#4257) Add search emails for group method in Registry NEW: (#4397) ConfigurationClient - added getOptionsDictRecursively method, add docs CHANGE: (#4551) Registry - change search DN properties logic *Framework CHANGE: (#4157) add possibility to work with Proxy Providers and "clean" proxy without voms/dirac extensions NEW: (#4257) Filter of duplicate mails in Notification FIX: (#4257) Optimize test in NotificationHandler FIX: (#4289) Avoid leaking std(in|out|err) file handles when running runsvdir CHANGE: (#4362) BundleDeliveryFramework uses list instead of tuple CHANGE: (#4410) (JSON) use of list instead of tuples for SecurityLog *WMS CHANGE: (#4362) SandboxStoreClient uses list instead of tuple CHANGE: (#4396) Removed outdated PilotMonitorAgent CHANGE: (#4410) (JSON) use of list instead of tuples for JobReport and Watchdog CHANGE: (#4416) As explained in #4412, Completed jobs is used as major status for jobs during completion while it is also used for jobs that have a pending request and require a further action before being Done or Failed. CHANGE: (#4416) Completing jobs are treated exactly like Running jobs, i.e. set Stalled if they didn't give sign of life, and are not set Failed directly as before. Jobs to be checked are those that had not sent a heartbeat within the stalledTime period, in order to reduce the number of jobs to check. NEW: (#4416) Created a file JobStatus.py for replacing the literal job statuses, and updated many modules using these new variables. Note that there are still many others to change, but could be done gradually... FIX: (#4451) StalledJobAgent - use int factor to the seconds time delta FIX: (#4451) ProxyDB - set UserName in setPersistencyFlag() if it is a first record in ProxyDB_Proxies FIX: (#4514) JobWrapper: try to send the failover requests BEFORE declaring the job status FIX: (#4514) JobAgent: do not override the job status after the job has finished *DMS CHANGE: (#4243) FileCatalog - standard components are loaded using ObjectLoader mechanism NEW: (#4279) FileCatalog - added VOMSSecurityManager class CHANGE: (#4279) FileCatalog - dropped PolicyBasedSecurityManager and VOMSPolicy classes CHANGE: (#4410) (JSON) correct casts for serialization for FTS CHANGE: (#4364) remove all the SE mangling in the DFC (SEPrefix, resolvePFN, SEDefinition) CHANGE: (#4423) FileMetadata - split getFileMetadataFields into getFileMetadataFields and _getFileMetadataFields CHANGE: (#4423) DirectoryMetadata - split getMetadataFields into getMetadataFields and _getMetadataFields NEW: (#4465) FileCatalog - a FileMetadata and DirectoryMetadata multi VO targeted plug-in wrappers for user metadata operations. FIX: (#4535) StorageElement - do not return negative values in getFreeDiskSpace() *RMS CHANGE: (#4410) (JSON) correct casts for serialization FIX: (#4517) When finalising the request, one should take into account the fact that if the job is Stalled one must find its previous status. This is fixed in ReqClient *Resources NEW: (#4157) ProxyProvider resources with implementation for DIRAC CA and PUSP proxy providers NEW: (#4276) IdProvider resources to represent external user identity providers FIX: (#4455) discrepancy in CE._reset() methods return value in different CE implementations NEW: (#4546) add Test_ProxyProviderFactory, docs *RSS CHANGE: (#4362) SQL query orders using list instead of tuples CHANGE: (#4410) (JSON) db ordering uses list instead of tuples FIX: (#4462) convert the dictionary keys to number, when it is required. CHANGE: (#4463) Dynamically load database handlers for ResourceStatus and ResourceManagement modules in preparation for a schema changes needed by multi-VO versions of these. *StorageManagement CHANGE: (#4410) (JSON) use lists instead of tuples *tests CHANGE: (#4279) use VOMSSecurityManager for integration instead of PolicyBasedSecurityManager/VOMSPolicy FIX: (#4284) DMS client tests to use DIRAC user name, rather than the local system user name NEW: (#4289) CI for unit/lint/integration testing with GitHub actions NEW: (#4289) Start using shell linting of the test scripts FIX: (#4289) Exit quickly if the installation fails to make it easier to find issues CHANGE: (#4410) call pytest directly instead of as a module FIX: (#4426) restored system test, moved Jenkins test to appropriate location CHANGE: (#4439) integrate test_MySQLDB to all_integration_server_tests *tests FIX: (#4469) ProxyDB integration tests fixed for the M2Crypto case FIX: (#4551) align ProxyDB test to current changes *docs NEW: (#4289) Document how to run integration tests in docker NEW: (#4551) add DNProperties description to Registry/Users subsection [v7r0p57] *Resources NEW: (#5067) RucioFileCatalog plugin based on Belle II implementation [v7r0p56] *Resources FIX: (#5119) HTCondorCE: Limit calls to actual cleanup (find and delete files on disk) to once per minute per SiteDirector, fixes #5118 CHANGE: (#5119) HTCondorCE cleanup: Run the DIRAC_ executable purge with -O3 and -maxdepth 1 to speed up the find [v7r0p55] *TS FIX: (#5109) DataRecoveryAgent: immediately skip transformations without jobs, prevents exception later on [v7r0p54] *Core FIX: (#5087) Undefined variable error when RabbitMQAdmin fails to connect *CS FIX: (#5090) VOMS2CSSynchronizer: add email information about a user joining groups for an additional VO, fix for #5037 FIX: (#5090) VOMS2CSSynchronizer: When a user leaves a second VO, only the membership in these groups is removed CHANGE: (#5090) VOMS2CSSynchronizer: If email of user is known, do not replace it with None [v7r0p53] *Interface FIX: (#5069) parseArguments strip arguments correctly [v7r0p52] *Core FIX: (#5058) (M2Crypto) fixes in proxy serial parsing [v7r0p51] *Configuration FIX: (#5037) VOMS2CSAgent: fix an issue in the agent that prevented users being added to a second VO if they were already existing in another VO [v7r0p50] *Core CHANGE: (#4927) Added MySQL LIKE query feature. [v7r0p49] *Core NEW: (#4967) introduce DIRAC_M2CRYPTO_SSL_METHODS to configure accepted protocols NEW: (#4967) introduce DIRAC_M2CRYPTO_SSL_CIPHERS to configure accepted ciphers [v7r0p48] CHANGE: (#4949) change default top-level bdii to cclcgtopbdii01.in2p3.fr (was lcg-bdii.cern.ch) *CS FIX: (#4943) Fix exception in VOMS2CSAgent for missing 'suspended' and other entries occurring if the credentials used to query the VOMS server have insufficient privileges to obtain this information [v7r0p47] *Core CHANGE: (#4930) Resolve location of etc/grid-security/vomses via env variable X509_VOMSES [v7r0p46] *DMS FIX: (#4908) FTS3Job: fix the check of isTapeSE for multiVO instances, fixes #4878 FIX: (#4908) FTS3Job: pass VO for fetchSpaceToken function to StorageElement [v7r0p45] *WMS FIX: (#4899) JobAgent: adding ExtraOptions as argument iff the executable is a DIRAC script *Resources CHANGE: (#4896) ARC CE: request all processors on the same node. [v7r0p44] *Core CHANGE: (#4873) default generated bashrc does not overwrite X509* variables if already set NEW: (#4873) Add DIRAC_X509_HOST_CERT and DIRAC_X509_HOST_KEY environment variables *WMS CHANGE: (#4836) JobWrapper: Add extraOptions only when the executable is a DIRAC script [v7r0p43] *RSS FIX: (#4871) SiteInspectorAgent: prevent the agent from getting locked if an exception occurs during enforcement FIX: (#4871) ElementInspectorAgent: prevent the agent from getting locked if an exception occurs during enforcement [v7r0p42] *Configuration CHANGE: (#4587) Added possibility to load more then one cfg file from DIRACSYSCONFIGFILE env variable [v7r0p41] *Core FIX: (#4845) Fix hashing SubjectName in X509Chain *FrameworkSystem FIX: (#4833) pilot update script, update global CS not the local one NEW: (#4840) Add option to retry to update hosts in dirac-admin-update-instance *WorkloadManagementSystem FIX: (#4838) SiteDirector - Flatten platform list avoid crash if CheckPlatform=True [v7r0p40] *docs FIX: (#4826) correct link to LHCb documentation [v7r0p39] NEW: (#4814) FileManager, FileCatalogDB - added repairFilesTable function to fix differencies between FC_Files and FC_FileInfo tables NEW: (#4814) FileCatalog CLI - rmdir command can perform recursive directory removal NEW: (#4814) FileCatalog CLI - repair command adapted to multiple repair operations [v7r0p38] *Configuration FIX: (#4807) --cfg option supports tilde FIX: (#4813) Show username when deleting users *FrameworkSystem NEW: (#4800) new tool dirac-admin-update-instance to update all servers of a dirac instance from command line NEW: (#4800) new tool dirac-admin-update-pilot to update version of pilot in CS from command line *DMS CHANGE: (#4807) FTS3 gives priority to disk replicas *Resources NEW: (#4807) CTA compatible Storage plugin FIX: (#4812) get SSH hostname from the CE config instead of the job ID to get the job output *tests FIX: (#4803) Use GitHub Container registry due to pull limit on dockerhub CHANGE: (#4806) Remove obsolete gitlab integration [v7r0p37] *TS CHANGE: (#4769) consider all sites when getting sites with storage (not only LCG sites with Tier0, 1 or 2 *WMS FIX: (#4769) dirac-wms-job-parameters: output was printed twice *docs NEW: (#4771) admin tutorial on how to install the WorkloadManagementSystem CHANGE: (#4780) mock the MySQLDB python module as we cannot install it in RTD any longer [v7r0p36] FIX: fixes from v6r22p35 patch release [v7r0p35] *Core FIX: (#4751) Do not read suspensionReason from VOMS as it's not used anywhere and needs special permissions to give this information FIX: (#4756) Fix for silly-formed XMLs from GOCDB *Framework CHANGE: (#4753) NotificationService: if avoidSpam=True the email is now sent immediately and cached for up to one hour, instead of being cached and then sent after up to an hour. *Resources FIX: (#4743) remove dots in xroot virtual username [v7r0p34] *Core NEW: (#4391) Make M2Crypto TLS implementation compatible with openssl 1.1 series *DataManagement FIX: (#4729) Allow no LFNs to be passed to getReplicas *RSS FIX: (#4722) GOCDBClient: fix for different forms of XML answer from GOC *Resources FIX: (#4725) create "task kwargs" with the required arguments in PoolCE before passing them to the ProcessPool FIX: (#4738) create chunks before using the Arc.JobSupervisor [v7r0p33] *Interfaces FIX: (#4713) Pass useCertificates to WMSClient in Dirac API *Resources NEW: (#4716) better way to discover the innerCESubmissionType (Pool CE) CHANGE: (#4716) SingularityCE: get info from Pilot3 pilot.json, if present *tests NEW: (#4716) added integration test for SingularityCE [v7r0p32] *Core FIX: (#4688) M2Crypto based protocol - better handling of timeout NEW: (#4710) add loadCRLFromFile to pyGSI implementation of X509CRL *WorkloadManagement NEW: (#4708) JobAgent - define DIRAC_JOB_PROCESSORS environment to number of processors allocated for the user job [v7r0p31] *WorkloadManagement FIX: (#4706) Crash getting available memory limits when MJF is not available [v7r0p30] *Core FIX: (#4698) Glue2: Fix crash in bdii2CSAgent, when ARC CEs do not have an ExecutionEnvironment *WorkloadManagementSystem FIX: (#4701) Ensure JobParameters.getNumberOfProcessors always returns an integer *RSS NEW: (#4691) GOCDB - interpret also downtimes with URLs different from the hostname *Resources CHANGE: (#4666) ARCComputingElement - enable GLUE2 queries on ARC CEs, which is now strongly recommended as GLUE1 publishing is stopped in ARC6. *docs NEW: (#4684) Added documentation for HTCondor/use full proxy length setting. [v7r0p29] *Core FIX: (#4683) M2Crypto - force bio free and shutdown when closing *Configuration CHANGE: (#4651) Add --cfg option for config files, deprecating old non-option version. *TransformationSystem FIX: (#4680) Bug fixed in updateFilterQueries. The bug was affecting only TS catalog interface. *Production FIX: (#4672) minor fix in ProdValidator FIX: (#4672) more detailed output added in dirac-prod-get-trans *Resources FIX: (#4673) hack in Slurm ResourceUsage to stop jobs running out of time on multi-processors resources [v7r0p28] *Core FIX: (#4642) M2Crypto closes the socket after dereferencing the Connection instance *DMS FIX: (#4644) Cancel FTS3 Operation if the RMS request does not exist *RMS NEW: (#4644) getRequestStatus returns ENOENT if the request does not exist *TS FIX: (#4647) TaskManager - hospital sites were not looked for correctly, which generated an exception FIX: (#4656) fix parsing of command line flags (e.g., -ddd) for dirac-transformation-archive/clean/remove-output/verify-outputdata *Interfaces CHANGE: (#4652) dirac-admin-add-host now inserts hosts into the ComponentMonitoring if the host is not yet known [v7r0p27] *Framework FIX: (#4639) MySQL - commit transaction when creating tables [v7r0p26] *Resources CHANGE: (#4626) Test_PoolComputingElement - increase delays for job submission CHANGE: (#4626) PoolComputingElement - fix the logic of number of processors evaluation FIX: (#4634) Add pilot.cfg symlink in SingularityCE for Pilot3 compatibility. FIX: (#4620) in getPilotOutput, returns errors when the HTCondor working directory is not available and condor_transfer_data fails *WMS NEW: (#4614) uploadToWebApp flag to disable the json upload for the CSToJson FIX: (#4614) Fix an absolute path in the CSToJSON NEW: (#4620) pop the pilot reference from the failedPilotOutput dictionary when maxRetryGetPilotOutput is reached NEW: (#4619) Watchdog - added DISABLE_WATCHDOG_CPU_WALLCLOCK_CHECK env variable FIX: (#4619) Watchdog - Split check methods for better control *Docs: CHANGE: (#4614) allow to generate only header/footer in the release notes if no PR were merged FIX: (#4635) add missing packages for the UBUNTU Linux Distribution that need to be installed before installing DIRAC NEW: (#4635) add useful commands for working with conda environments CHANGE: (#4622) diracdoctools - No longer mock numpy and matplotlib FIX: (#4619) added doc about DIRACSYSCONFIG environment variable [v7r0p25] Fixes from v6r22p29 patch *Core FIX: (#4615) M2SSLTransport does not catch all the exceptions FIX: (#4615) M2SSLTransport calls parent class when renewing context CHANGE: (#4615) default to split handshake NEW: (#4617) Added test for profiler.py *WMS FIX: (#4617) Watchdog - use 2 flags for profiler: withChildren and withTerminatedChildren [v7r0p24] *Core FIX: (#4584) M2SSLTransport catch exceptions and convert them into S_ERROR CHANGE: (#4584) M2SSLTransport: allow to do the SSL handshake in threads CHANGE: (#4584) Do not query the Registry when doing the handshake *WMS CHANGE: (#4587) Check LFN InputSandbox separately from InputData NEW: (#4593) option to only retry to get pilot outputs a limited number of times CHANGE: (#4594) SandboxStoreClient - remove direct access to SandboxMetadataDB by default CHANGE: (#4594) JobSanity - instantiate SandboxStoreClient with direct access to SandboxMetadataDB FIX: (#4601) JobAgent: check if there are arguments *Resources CHANGE: (#4593) retrieve pilot output paths in getJobOutput using the pilot IDs instead of calling condor *Interfaces Change: (#4594) Dirac - instantiate SandboxStoreClient without smdb=False which is now a default *docs FIX: (#4598) Use also time in addition to date to get list of PRs since last tag [v7r0p23] *Core FIX: (#4580) The CPU for the jobWrapper process group was incorrectly calculated as it was not including former children of the JobWrapper. *Framework FIX: (#4570) MonitoringCatalog only prints an error when there is really one *WMS NEW: (#4576) add a workDir to PilotCStoJSONSynchronizer FIX: (#4591) Fix exception when calling HTCondorCE killJob, used when killing pilots with dirac-admin-kill-pilot for example. Fixes #4590 *DMS FIX: (#4591) allow dirac-dms-protocol-matrix be run for non LHCb VOs *RMS CHANGE: (#4573) Add include from ConfigTemplate to CleanReqDBAgent, RequestExecutingAgent, ReqManager CHANGE: (#4573) align defaults in ConfigTemplate with those in the Code *Resources FIX: (#4588) HTCondorCE cleanup only the pilots files related to the CE *Docs: NEW: (#4571) the DiracDocTools are now also compatible with python3 NEW: (#4571) added ".readthedocs.yml" config [v7r0p22] *Core: CHANGE: (#4554) dirac-install: remove empty lines between options in dirac-install --help *Configuration CHANGE: (#4563) move documentation to ConfigTemplate, Module docstrings *Accounting CHANGE: (#4564) Move documentation about agent and services configuration to module docstrings, ConfigTemplate *Docs CHANGE: (#4554) Small restructuring in the DeveloperGuide: merge sections on testing, add note about generating command references NEW: (#4554) DeveloperGuide: add section about DIRACOS and link to diracos.readthedocs.io CHANGE: (#4560) allow release notes to be empty in dirac-docs-get-release-notes *Tests FIX: (#4560) DataManager tests check VO dynamically [v7r0p21] *Core CHANGE: (#4506) remove rst2pdf from dirac-create-distribution-tarball *Framework CHANGE: (#4536) Don't loop forever if a query does not work in the MonitoringCatalog [v7r0p20] *Core FIX: (#4531) Mail.py - add SMTP parameters *DMS CHANGE: (#4528) FTS3 - failoverTransfer sleep & retries in case of FC unavailability *Resources NEW: (#4529) WLCGAccountingHTTPJson occupancy plugin *TS CHANGE: (#4525) clean the DataFiles table when cleaning a Transformation [v7r0p19] *DMS CHANGE: (#4505) explicitly delegate the proxy to FTS3 with a configurable lifetime NEW: (#4519) S3 storage support *RSS FIX: (#4520) correct verbosity (avoid non-necessary warnings) *docs NEW: (#4520) Minimal documentation on Bdii2CSAgent and GOCDB2CSAgent [v7r0p18] *Interfaces CHANGE: (#4502) run jobs locally: use $DIRAC for $DIRACROOT *Docs NEW: (#4513) add recommonmark extension to allow use of markdown files in the documenation [v7r0p17] *WMS FIX: (#4496) PilotCS2Json: fix writing of local pilot.json file, which is used to calculate the hash FIX: (#4496) PilotCS2Json: write checksum file in text mode *DMS FIX: (#4476) StorageElementHandler: removed ignoreMaxStorageSize flag from getTotalDiskSpace and getFreeDiskSpace *Resources FIX: (#4499) Slurm Resource Usage [v7r0p16] *Core CHANGE: (#4482) TimeLeft modules become ResourceUsage and inherit from an abstract module called ResourceUsage NEW: (#4482) TimeLeft/SLURMResourceUsage module to get resource usage from a SLURM installation CHANGE: (#4482) Move TimeLeft from Core/Utilities to Resources/Computing/BatchSystems *WMS NEW: (#4493) Add checksum calculation for PilotCSToJsonSynchroniser NEW: (#4493) Add checksum checks to PilotWrapper FIX: (#4493) Fix download checks in PilotWrapper in case the webserver does not return 404 but not the expected file content either. *Resources FIX: (#4482) LocalCE: proxy submission and getJobStatus FIX: (#4482) Slurm getJobStatus() making use of sacct FIX: (#4466) Stomp reconnection only reconnect the connection that dropped [v7r0p15] *Accounting NEW: (#4464) StorageOccupancy accounting *RSS NEW: (#4464) FreeDiskSpaceCommand used to fill the StorageOccupancy accounting CHANGE: (#4464) FreeDiskSpaceCommand can clean the from not-anymore-existing SEs *WMS CHANGE: (#4471) Optimizer: treat input sandboxes uploaded as LFNs ad Input Data [v7r0p14] *Framework CHANGE: (#4458) Remove LHCb specific code used to install cx_Oracle *Core CHANGE: (#4460) agents exit with sys.exit(2) in case of errors *Resources CHANGE: (#4460) InProcess CE correctly reports number of processors available *WMS FIX: (#4454) PilotWrapper: try to untar also with system tar [v7r0p13] *Core FIX: (#4448) SubProcess: rewritten function for getting child PIDs *ResourceStatusSystem NEW: (#4419) Backported GGUSTicketsPolicy from LHCb CHANGE: (#4419) removed dangerous script dirac-rss-policy-manager *tests NEW: (#4429) Allow extensions to extend the continuous integration tests *docs NEW: (#4431) dirac-doc-get-release-notes can take a sinceLatestTag option, header and footer messages, and can create github/gitlab release [v7r0p12] *WMS FIX: (#4432) Better logging for WMS Optimizers NEW: (#4435) PilotCS2JSONSynchronizer: added options pilotRepoBranch and pilotVORepoBranch *DMS FIX: (#4436) dirac-dms-clean-directory: also works on single empty directory, fixes #4420 *Framework FIX: (#4436) dirac-install-component: fix bug prohibiting use of the -m/--module parameter *tests NEW: (#4429) Allow extensions to extend the continuous integration tests NEW: (#4433) added README.rst for every subdirectory *docs FIX: (#4438) update notes for dirac-distribution (add extensions) [v7r0p11] *Core FIX: (#4411) Make dirac-install tool python 3 compatible NEW: (#4409) Allow --dirac-os-version argument to dirac-install.py to be a path or URL to a tarball [v7r0p10] *Core FIX: (#4394) RequestHandler - fix log output for the case where the ActionTuple is actually a string FIX: (#4394) AuthManager - manage the case where ExtraCredentials are in a form of a list NEW: (#4379) dirac-install accepts userEnvVariables switch, adds user-requested env variables to *rc* files *WMS FIX: (#4404) Fix the matching delay functionality, fixes #2983 NEW: (#4403) JobCleaningAgent: Add possibility to remove HeartBeatLoggingInfo before jobs are completely removed. FIX: (#4394) SiteDirector - do not add downloading files ti the pilot if Pilot3 flag is False FIX: (#4390) Watchdog: get the CPU consumed by children processes too FIX: (#4370) SiteDirector: Pilot3 option easier to interpret CHANGE: (#4338) getStatus method in ARC CE now uses a JobSupervisor to get status of multiple pilots at once. CHANGE: (#4338) SiteDirector.updatePilotStatus has been parallelized using threads. NEW: (#4338) AvailableSlotsUpdateCycleFactor is a parameter part of the configuration allowing to control the rate of the update of the available slots in the queues. FIX: (#4338) Revert to queueCECache being an object attribute to fix the CEs instantiation. *RMS CHANGE: (#4400) ReqDB: Add pool_recycle=3600 parameter for sql engine setup, might prevent occasional "Mysql Server has gone away" errors *Resources FIX: (#4380) ComputingElement classes: use GridEnv if present CHANGE: (#4360) in HTCondor CEs, the pilot stamps are now used to retrieve outputs and logging info *DMS CHANGE: (#4400) FTS3DB: Add pool_recycle=3600 parameter for sql engine setup, might prevent occasional "Mysql Server has gone away" errors *Docs CHANGE: (#4402) dirac-docs-get-release-notes.py does not require a GITLABTOKEN [v7r0p9] *Core CHANGE: (#4302) Simplify and convert command wrapper scripts to bash. Also removed some obsolete MacOS code. FIX: (#4361) restore the possibility to set a global timeout for a Client *DMS CHANGE: (#4353) FTS3 persistOperation method checks that the caller is allowed to do so CHANGE: (#4353) FTS3Manager: do not expose updateJobStatus and updateFileStatus on the service CHANGE: (#4353) FTS3 operation is canceled if the matching request is canceled CHANGE: (#4353) when a source file does not exist, defunct the FTS3File associated CHANGE: (#4353) use the JEncode serializer instead of the custom FTS3Serializer CHANGE: (#4353) cancel an FTS3Job together with its associated FTS3Files if the job is not found on the server *RSS CHANGE: (#4336) remove a call to shifterProxy configuration which would not work for multi VO Dirac *docs CHANGE: (#4378) using docker images from docker hub [v7r0p8] *Configuration CHANGE: Let update the config, even if the Pilot info is not in the CS [v7r0p7] *Core CHANGE: {bash,tchs,diracos}rc set the PYTHONPATH to only dirac (ignore existing PYTHONPATH) CHANGE: dirac-configure might work without the need of a proxy CHANGE: the timeout of an RPC call is always updated [v7r0p8] *Configuration CHANGE: Let update the config, even if the Pilot info is not in the CS [v7r0p7] *Core CHANGE: {bash,tchs,diracos}rc set the PYTHONPATH to only dirac (ignore existing PYTHONPATH) CHANGE: dirac-configure might work without the need of a proxy CHANGE: the timeout of an RPC call is always updated [v7r0p6] *Core NEW: Add environment.yml file for preparing an environment with conda CHANGE: Use subprocess32 if possible for significantly better performance FIX: TypeLoader should not be a singleton as it has a cache *FrameworkSystem FIX: Avoid leaking std(in|out|err) file handles when running runsvdir *WorkloadManagementSystem NEW: To activate the pilot logging mechanism the following option must be present in CS: Pilot/PilotLogging with a value set to "true", "yes" or "y". *Tests NEW: CI for unit/lint/integration testing with GitHub actions NEW: Start using shell linting of the test scripts FIX: Exit quickly if the installation fails to make it easier to find issues *Docs NEW: Document how to run integration tests in docker CHANGE: The options in the command reference sections: 'ignore' and 'scripts', are replaced by 'exclude' and 'manual' respectively. The entries in 'exclude' will now reject scripts that are otherwise picked up by the 'patterns'. Entries in 'manual' will be added to the indexFile, but their rst file has to be provided by hand. Fixes #4345 [v7r0p5] FIX: changes from v6r22p15 included [v7r0p4] *Core FIX: (#4337) remove unexisting import from dirac-info *WMS CHANGE: (#4317) dealing with min and max number of processors from the job *docs CHANGE: (#4317) added WMS admin doc (job state machine) [v7r0p3] *Core FIX: (#4298) install_site.sh : set UpdatePilotCStoJSONFile=False for initial installations *WMS FIX: (#4294) JobState - restored logic of having database clients at the class level *TS CHANGE: (#4308) Rename Operations option Productions/ProductionFilesMaxResetCounter to Transformations/FilesMaxResetCounter *Resources, WMS, tests CHANGE: (#4295) save the number of processors as jobLimits in pilot cfg *tests CHANGE: (#4291) removed pilot2 tests *docs NEW: (#4309) Added how to create a dedicated dirac file catalog [v7r0p2] *Accounting FIX: (#4290) Allow longer user and site names. *WMS NEW: (#4287) dirac-wms-get-wn and dirac-wms-pilot-job-info scripts (ported from LHCbDIRAC) *docs CHANGE: (#4266) the AdministratorGuide has been restructured [v7r0p1] FIX: Removing some deprecated codes [v7r0] FIX: (#3962) use print function instead of print statement FIX: (#3962) remove the usage of the long suffix to distinguish between long and int FIX: (#3962) convert octal literals to new syntax NEW: (#3962) Add pylint test to check python 3 compatibility *Configuration CHANGE: (#4249) The flag UpdatePilotCStoJSONFile has been moved to Operations/[]/Pilot section NEW: (#4255) Resources - added getStorageElements() method *Framework CHANGE: (#4180) Removed local MySQL handling (DIRACOS won't have install it) CHANGE: (#4180) Removed setup of old portal *Accounting CHANGE: (Multi)AccountingDB - Grouping Type and Object loader together with the MonitoringSystem ones. *WorkloadManagementSystem NEW: Add JobElasticDB.py with getJobParameters and setJobParameter methods to work with ElasticSearch (ES) backend. NEW: Add gJobElasticDB variable and indicates the activation of ES backend. CHANGE: Modify export_getJobParameter(s) to report values from ES if available. CHANGE: (#3748) Reduced (removed, in fact) interactions of Optimizers with JobParameters, using only OptimizerParameters NEW: (#3760) Add Client/JobStateUpdateClient.py NEW: (#3760) Add Client/JobManagerClient.py CHANGE: (#3760) Use the above Client classes instead of invoking RPCClient() NEW: Added ES backend to WMSAdministratorHandler to get JobParameters. NEW: Added separate MySQL table for JobsStatus in JobDB, and modified code accordingly. CHANGE: ElasticJobDB.py: Modify setJobParameter method to register JobAttributes like Owner, Proxy, JobGroup etc. CHANGE: ElasticJobDB.py: added getJobParametersAndAttributes method to retrieve both parameters and attributes for a given JobID. CHANGE: Pilot Watchdog - using python UNIX services for some watchdog calls CHANGE: (#3890) JobState always connects to DBs directly CHANGE: (#3890) remove JobStateSync service CHANGE: (#3873) Watchdog: use Profiler instead of ProcessMonitor NEW: (#4163) SiteDirectors send by default Pilot3 (flag for pilot2 is still kept) FIX: (#4163) removed unicode_literals (messes up things with DEncode) NEW: (#4224) PilotWrapper can get the Pilot files from a list of locations CHANGE: (#4224) PilotWrapper is compatible with several python versions NEW: (#4260) New dirac-wms-match command to test a given job for matching computing resources NEW: (#4260) QueueUtilities - utilities relevant to queues manipulation FIX: (#4265) urllib.urlopen() call with and without the 'context' parameter in Utilities/PilotWrapper.py according to the version of urllib instead of the python version CHANGE: (#4244) Modified flag for using the JobParameters on ElasticSearch database *Core NEW: (#3744) Add update method to the ElasticSearchDB.py to update or if not available create the values sent from the setJobParameter function. Uses update_by_query and index ES APIs. CHANGE: (#3744) generateFullIndexName() method made static under the ElasticSearchDB class. CHANGE: (#3744) removed unused/outdated stuff from Distribution module FIX: check for empty /etc/grid-security/certificates dir NEW: (#3842) Add createClient decorator to add wrapper for all export_ functions automatically NEW: (#3678) dirac-install can install a non released code directly from the git repository FIX: (#3931) AuthManager - modified to work with the case of unregistered DN in credDict FIX: (#4182) Add CountryName OID and be more permissive in the String type for decoding VOMSExtensions FIX: (#4211) change orders of @deprecated and @classmethod decorators in X509Chain NEW: (#4229) dirac-install defines XRD_RUNFORKHANDLER environment variable *TransformationSystem NEW: (#4124) InputDataQuery and OutputDataQuery parameters can be set for Transformations before they are added to the transformation system. Solves #4071 CHANGE: (#4238) TransformationDB.getTransformationMetaQuery returns ENOENT if no meta query exists FIX: (#4238) InputDataAgent: silence warnings about transformations not having a input data query *ProductionManagement NEW: (#3703) ProductionManagement system is introduced *Interfaces CHANGE: (#4163) removed deprecated methods from Dirac.py *Resources FIX: (#4229) add proxy location as a virtual user for xroot urls FIX: (#4234) future import in executeBatch script that prevented the pilot deployment on SSH CE, fixes #4233 FIX: (#4231) SE __executeMethod: only pass protocols parameter to getTransportURL NEW: (#4255) New dirac-resource-info command to display computing and storage resources available to a given VO *ResourceStatusSystem CHANGE: (#4177) use the ObjectLoader for Clients, in PEP *Monitoring FIX: MonitoringReporter - make processRecords() method thread safe, fixes #4193 *tests NEW: Add ChangeESFlag.py script to modify useES flag in dirac.cfg. To be integrated with Jenkins code. CHANGE: (#3760) Use the above Client classes instead of invoking RPCClient() NEW: (#3744) Added performance tests for ES and MySQL for WMS DB backends NEW: (#3744) Added miniInstallDIRAC function for Jenkins jobs FIX: (#4177) restored test of JobsMonitor().JobParameters NEW: (#4224) added a test for PilotWrapper NEW: (#4244) integration test for JobParameters on ElasticSearch database *Docs FIX: Better dirac.cfg example configuration for web NEW: (#3744) Add WMS documentation in DeveloperGuide/Systems NEW: Added script (docs/Tools/UpdateDiracCFG.py) to collate the ConfigTemplate.cfg files into the main dirac.cfg file CHANGE: (#4110) updated basic tutorial for CC7 instead of SLC6. NEW: (#4170) added Production system documentation CHANGE: (#4224) Pilot 3 is the default NEW: (#4244) Added a few notes on using the JobParameters on ElasticSearch database [v6r22p35] *WMS FIX: (#4759) exclude fuse from df (watchdog checks) [v6r22p34] *DMS FIX: (#4748) FTS3Agent - Rotate FTS3Operations list to fetch all. Fixes #4727 [v6r22p33] *Core NEW: (#4710) add loadCRLFromFile to pyGSI implementation of X509CRL [v6r22p32] *Core Change: (#4665) ElasticSearchDB - existing index function is modified to add argument option of document id NEW: (#4665) ElasticSearchDB - update() function is added. *Resources CHANGE: (#4676) ARCComputingElement - reserve the number of cores as defined in the NumberOfProcessors configuration parameter FIX: (#4676) ARCComputingElement - use CEQueueName configuration parameter if defined [v6r22p31] *DMS FIX: (#4646) Print error from dirac-dms-add-file if input LFN list file is missing. *RMS FIX: (#4657) RequestDB - fix getRequestCountersWeb error in DIRACOS [v6r22p30] *DataManagementSystem FIX: (#4627) Throw an error if LFN contains '//' [v6r22p29] *WorkloadManagementSystem CHANGE: (#4603) Updated to customize the JobType of user jobs in JobDescription and JobManifest *DataManagementSystem CHANGE: (#4618) dirac-dms-remove-catalog-files and dirac-dms-remove-catalog-files use Operations/.../DataManagement/AllowUserReplicaManagement to allow users perform direct File Catalog operations [v6r22p28] *TS FIX: (#4569) TransformationClient - allow "LFN" condition as a string to be compatible with the service [v6r22p27] *Monitoring CHANGE: (#4543) Allow to retrieve multiple variables from Monitoring DB. Add functions to calculate efficiency. *TS FIX: (#4550) infinite loop when a file was requested an it was not in the TS CHANGE: (#4550) show headers in Utilities sub logger (but this is not effective yet) [v6r22p26] *Resources FIX: (#4518) HTCondorCE - added missing multicore option *Transformation FIX: (#4523) when getTransformationFiles was called with a large list of LFNs, the DB couldn't cope. This fix truncates the list of LFNs in chunks such that each call to the DB is fast enough. [v6r22p25] *Framework FIX: (#4503) SystemLoggingDB: reduce too long messages to 255 characters (see also #1780 ?) *WMS FIX: (#4503) JobLoggingDB: Limit StatusSource to 32 characters *TS CHANGE: (#4503) TransformationDB.Transformations TransformationGroup column increased to 255 characters (ALTER TABLE Transformations MODIFY TransformationGroup VARCHAR(255) NOT NULL default 'General';) *Resources FIX: (#4511) Moved _prepareRemoteHost in SSHComputingElement [v6r22p24] *Core FIX: (#4477) Try to untar using system tar in case tarfile module is failing [v6r22p23] *DMS CHANGE: (#4472) DM.getFile only checks metadata of files with replicas [v6r22p22] *Framework CHANGE: (#4457) Web portal compilation is done during the DIRAC distribution, it is not required to compile during the installation *RSS FIX: (#4452) PublisherHandler - fix UnboundLocalError [v6r22p21] *Core FIX: (#4442) If no port is set in the CS for ES assume that the URL points to right location *RSS FIX: (#4441) DowntimeCommand - fix typo that was preventing to update expired or deleted downtimes from RSS downtime cache. *ConfigurationSystem FIX: (#4447) remove logging from inside getConfigurationTree *MonitoringSystem FIX: (#4447) add check if MQ is setup for Monitoring CHANGE: (#4443) Read the IndexPrefix for the CS and use this index prefix when creating ES indices, if not given use the name of the setup [v6r22p20] *Core CHANGE: (#4424) SocketInfo.py - DEFAULT_SSL_CIPHERS updated following the issue #4393 *WorkloadManagement FIX: (#4440) ServerUtils.py - server name in ServerUtils.getPilotAgentsDB() *Transformation FIX: (#4434) TaskManager.py - fix bug in finding the settings for Clinics in the hospital [v6r22p19] *Framework CHANGE: (#4415) ComponentInstaller - do not start runsvdir if not needed *Resources FIX: (#4414) Enable setting of WaitingToRunningRatio from Global CE Defaults, fixes #4368 *DMS CHANGE: (#4413) FTS3Agent - change proxy lifetime of FTS3Agent from 2 to 12 hours [v6r22p18] *TS CHANGE: (#4331) TaskManager - give possibility to run jobs for different problematic productions at different Hospital sites [v6r22p17] CHANGE: Just update the Web version [v6r22p16] *Core CHANGE: remove many calls to gLogger.debug() of low level DB modules (in particular MySQL.py) that may slowdown services *FrameworkSystem FIX: PRIVATE_LIMITED_DELEGATION role can download its own proxies *ResourceStatusSystem FIX: PublisherHandler: Convert set to list, because set can not be serialized using DEncode. [v6r22p15] *Resources NEW: GFAL2_StorageBase: Disable GRIDFTP SESSION_REUSE by default. It can be enabled via an environment variable export DIRAC_GFAL_GRIDFTP_SESSION_REUSE=True. This export should be added in server bashrc files. [v6r22p14] *Framework FIX: (#4318) NotificationHandler: fix name of initializeHandler function so the handler is properly initialized and the periodicTasks are created CHANGE: (#4325) InstalledComponentsDB.addInstalledComponents: Instead of creating a new Host entry if only the CPU model changed, update the CPU field in the DBCHANGE: InstalledComponentsDB.addInstalledComponents: Instead of creating a new Host entry if only the CPU model changed, update the CPU field in the DB *WMS FIX: (#4329) Fix exception for PilotManager or PilotStatusAgent: itertems -> iteritems *DMS FIX: (#4327) avoid introducing inconsistencies in the DB due to the FTS inconsistencies *Resources CHANGE: (#4313) increase the SE logging level for plugin errors *TS CHANGE: (#4315) DataRecoveryAgent: Tweak information printout for email formatting; clarify use of default values; reduce logging verbosity [v6r22p13] *Resources FIX: (#4299) CREAMComputingElement - possibility to defined CEQueueName to be used in the pilot submission command CHANGE: (#4297) SingularityCE: Delete workDir at end of job by default. FIX: (#4297) SingularityCE: Always stop proxy renewal thread at end of job. *TS NEW: (#4301) DataRecoveryAgent to perform and apply consistency checks for transformations NEW: (#4301) dirac-transformation-recover-data : script to manually run consistency checks on individual transformations, for debugging and testing of the DataRecoveryAgent [v6r22p12] FIX: fixes from v6r21p16 [v6r22p11] *Interfaces FIX: (#4277) Dirac.py - fix error handling in getJobParameters() [v6r22p10] *Resources FIX: (#4271) StorageElement - loadObject of occupancyPlugin is called with the path to OccupancyPlugins [v6r22p9] *Core FIX: (#4269) Workflow - revert changes introduced in #4253 [v6r22p8] *WMS FIX: (#4256) handle empty results of getJobParameter in StalledJobAgent *Resources NEW: (#4223) Storage - occupancy plugin for WLCG standard accounting JSON FIX: (#4259) SingularityComputingElement - ensure lcgBundle is installed in container if LCGBundleVersion is set. [v6r22p7] *Core FIX: (#4253) Workflow - accept unicode file path to workflow XML *WMS NEW: (#4205) Statistics of Pilot submission is sent to Accounting System by SiteDirector. FIX: (#4250) use host credentials to query the SandboxMetadataDB for async removal *Resources CHANGE: (#4242) DFC SEManagerDB: only acquire the lock if the cache needs to be modified FIX: (#4251) preamble attribute was missing in SSHBatchComputingElement causing an error in _submitJobHost() *DMS FIX: (#4248) SEManager correct order of calls at init [v6r22p6] *Core CHANGE: (#4203) A VO with diracos extension, must be able to install the main diracos from DIRAC repository. *TS CHANGE: (#4218) do not set Job type 'hospital' [v6r22p5] *WMS CHANGE: (#4217) Sandbox: Adding OwnerDN and OwnerGroup for DelayedExternalDeletion *DMS CHANGE: (#4202) relax permissions on getLFNForGUID *Resources FIX: (#4200) re-allow the use of the gsiftp cache in the SE CHANGE: (#4206) Storage - improved treatment for the case without SpaceToken in SpaceOccupancy [v6r22p4] *Core NEW: (#4181) Allow to install an extension of DIRACOS *Docs FIX: (#4187) Create a TransformationAgent Plugin -- more concrete descriptions for adding a plugin to AllowedPlugins [v6r22p3] *WMS CHANGE: (#4175) added function to get the number of Processors, using it in dirac-wms-get-wn-parameters (invoked by the pilots) CHANGE: (#4175) changed port of PilotManager from 9129 to 9171 *docs CHANGE: (#4174) update WebApp administrator docs CHANGE: (#4175) removed mentions of MPIService [v6r22p2] *Core FIX: (#4165) $Id$ keyword must be replaced only at the beginning of the file *RSS FIX: (#4169) PublisherHandler fix (UnboundLocalError) *Docs NEW: (#4167) /Operations/DataManagement - Added a link to the documentation on "Multi Protocol" [v6r22p1] *WMS NEW: (#4147) added option for specifying a LocalCEType (which by default is "InProcess") *Interfaces NEW: (#4146) added setNumberOfProcessors method to Job() API *Resources NEW: (#4159) add dav(s) in the protocol lists of GFAL2_HTTPS *tests NEW: (#4154) Using variable for location of INSTALL_CFG_FILE (useful for extensions) [v6r22] *WorkloadManagementSystem NEW: (#4045) PilotsHandler service (to interact with PilotAgentsDB) CHANGE: (#4049) Pilot wrapper for pilot3: download files at runtime CHANGE: (#4119) removed last bit looking into /Registry/VOMS/ section CHANGE: (#4119) VOMS2CSAgent: mailFrom option is invalid, use MailFrom instead FIX: (#4074) fixed PilotManager service name in ConfigTemplate.cfg FIX: (#4100) use CAs to upload the pilot files to a dedicated web server using requests FIX: (#4106) fixes for logging messages for Matcher FIX: (#4106) fixes for logging messages for Optimizers NEW: (#4136) added DIRACOS option (for SiteDirectors) *ConfigurationSystem NEW: (#4053) VOMS2CSSynchronizer/VOMS2CSAgent - enable automatic synchronization of the Suspended user status with the VOMS database CHANGE: (#4113) VOMS2CSSynchronizer: considering the case when no email is provided by VOMS *Core NEW: (#4053) AuthManager - interpret the Suspended user status considering suspended users as anonymous visitors CHANGE: (#4053) The use of CS.py module is replaced by the use of Registry.py and CSGlobals.py *Interfaces CHANGE: (#4098) removed dirac-admin-get-site-protocols.py as it could give potentially wrong results (use dirac-dms-protocol-matrix instead) *Resources NEW: (#4142) enable to get SE occupancy from plugin NEW: (#4142) add occupancy plugin to retrieve the info from BDII CHANGE: (#4142) GFAL2_SRM2Storage.getOccupancy() calls parent if SpaceToken is not given *ResourceStatusSystem CHANGE: clients: using DIRAC.Core.Base.Client as base CHANGE: (#4098) SiteInspectorAgent runs only on sites with tokenOwner="rs_svc" CHANGE: (#4098) remove SRM dependencies CHANGE: (#4136) downtimeCommand will use the GOCServiceType only for SRM SEs FIX: (#4139) only take the first endpoint for the SpaceOccupancy *DataManagementSystem CHANGE: (#4136) Moved methods from ResourceStatusSystem/CSHelpers to DMSHelpers CHANGE: (#4138) FTS3 is now the default *docs NEW: (#4099) Instructions about setting up the DIRAC web server for pilot3 CHANGE: (#4119) added note on MultiProcessor jobs preparation *test FIX: (#4119) Not lhcb but dteam (for DIRAC certification) FIX: (#4139) client_dms.sh not lhcb specific CHANGE:(#4140) adapt transformation certification tests to dteam VO [v6r21p16] *Resources FIX: (#4292) SSHComputingElement - define X509_USER_PROXY in case of gsissh access *WMS FIX: (#4292) SiteDirector - do not use keyword arguments when making setPilotStatus call [v6r21p15] *WMS CHANGE: (#4214) Add an argument to the constructor of SandboxStoreClient for using in scripts that cannot use the DB directly *DMS NEW: (#4171) ArchiveFiles Request Operation: to create a tarball out of a list of LFNs NEW: (#4171) CheckMigration Request Operation to hold the request progress until the attached LFNs are migrated to tape NEW: (#4171) FCOnlyStorage StorageElement plugin to register LFNs without physical replica to conserve LFN metadata when they are archived, for example NEW: (#4171) dirac-dms-create-archive-request command to create a request to archive a list of LFNs and remove their physical copies NEW: (#4171) dirac-dms-create-moving-request command to move LFNs from a to b with optional "CheckMigration" step. as it uses the ReplicateAndRegister operation, transfer via FTS is also possible FIX: (#4171) FileCatalogClient: add "addFileAncestors" to list of "write functions" [v6r21p14] *DMS FIX: (#4192) dirac-dms-clean-directory correct usage message for list of arguments FIX: (#4192) dirac-dms-clean-directory now properly prints error messages FIX: (#4192) dirac-dms-clean-directory will now also clean empty directories FIX: (#4192) FileCatalog.DirectoryMetadata: prevent error when removeMetadataDirectory is called on empty list of directories, triggered when calling removeDirectory on non-existing directory FIX: (#4192) FileCatalog.DirectoryTreeBase: prevent maximum recursion depth exception when calling remove directory with illegal path *Resources CHANGE: (#4191) Storages: catch specific DPM/Globus error code when creating existing directory [v6r21p13] *RSS FIX: (#4173) only use the hostname of FTS servers in the RSS commands [v6r21p12] *WMS FIX: (#4155) JobDB.getAttributesForJobList: Return S_ERROR if unknown attributes are requested. Instead of potentially returning garbage a clear error message is returned. [v6r21p11] *Transformation FIX: (#4144) fixed a logic bug in counting numberOfTasks in MCExtension which is expected to limit the total number of tasks for MC transformations *Accounting FIX: (#4151) In AccountingDB.insertRecordThroughQueue fix bad dictionary key "0K" [v6r21p10] *Core FIX: (#4133) dirac-install: correct location for ARC plugins with DIRACOS *WMS CHANGE: (#4136) JobStateUpdateHandler - restoring the job status to Running after hearbeat *Docs NEW: (#4134) Added /Operations/DataManagement parameters for protocols [v6r21p9] *Core FIX: (#4130) correct symlinks in dirac-deploy [v6r21p8] *WMS CHANGE: (#4111) Better logging in JobWrapper CHANGE: (#4114) JobScheduling - allow both Tag and Tags option in the job JDL *DMS FIX: (#4101) FileManagerBase - use returned ID:LFN dict instead of the LFN list. Fixes the bug in getReplicasByMetadata reported in #4058 *TransformationSystem FIX: (#4112) TaskManager.py - testing if the request ID is correct was not done properly, test the numerical value [v6r21p7] *Core FIX: (#4076) A certain module like WebAppDIRAC must be checked out from the code repository once. *Resources FIX: (#4078) Fix the exception when StorageElement objects are created with a list of plugins *SMS NEW: (#4086) StageMonitorAgent: new option StoragePlugins to limit the protocols used to contact storagelements for staged files. *WMS FIX: (#4093) better logging from services *TS CHANGE: (#4095) ConfigTemplate for RequestTaskAgent now contains all options CHANGE: (#4096) the Broadcast TransformationPlugin no longer requires a SourceSE to be set. If it is set, the behaviour is unchanged CHANGE: (#4096) dirac-transformation-replication: change default pluging back to Broadcast (reverts #4066) *Docs: CHANGE: (#4095) AdministratorGuide install TS tutorial: added options MonitorFiles and MonitorTasks for TaskAgents [v6r21p6] *CS FIX: (#4064) Fix exception when calling dirac-admin-add-shifter with already existing values *Core NEW: (#4065) getIndexInList utility in List.py *Resources NEW: (#4065) add a SpaceReservation concept to storages NEW: (#4065) add a getEndpoint method to StorageBase *RSS CHANGE: (#4065) CSHelpers.getStorageElementEndpoint returns the endpoint or non srm protocol CHANGE: (#4065) add the SpaceReservation to the FreeDiskSpaceCommand result *TS FIX: (#4066) The dirac-transformation-replication script will now create valid transformations given only the required arguments. Instead of the 'Broadcast' plugin, the 'Standard' plugin is created if not SourceSE is given. If a value for the plugin argument is given, that will be used. *docs CHANGE: (#4069) DIRAC installation procedure is updated taking account DIRACOS CHANGE: (#4094) Pilots3 options: moved to /Operation/Pilot section [v6r21p5] *Core NEW: (#4046) allow install_site to install DIRACOS FIX: (#4047) dirac-deploy-scripts uses correct regex to find scripts NEW: (#4047) dirac-deploy-scripts can use symplink instead of wrapper CHANGE: (#4051) use debug level for logs in the ProcessPool *RequestManagementSystem CHANGE: (#4051) split log messages *ResourceStatusSystem FIX: (#4050) fix reporting from EmailAgent CHANGE: (#4051) split log messages in static and dynamic parts *Docs CHANGE: (#4034) Add magic runs to setup DIRAC in example scripts, so they work out of the box. NEW: (#4046) add a tuto for setting up a basic installation NEW: (#4046) add a tuto for setting up two Dirac SEs NEW: (#4046) add a tuto for setting up the DFC NEW: (#4046) add a tuto for managing identities NEW: (#4046) add a tuto for setting up the RMS NEW: (#4046) add a tuto for doing DMS with TS *ConfigurationSystem CHANGE: (#4044) dirac-configure: forcing update (override, in fact) of CS list *WorkloadManagementSystem FIX: (#4052) SiteDirector - restore the logic of limiting the number of pilots to submit due to the WaitingToRunningRatio option FIX: (#4052) Matcher - if a pilot presents OwnerGroup parameter in its description, this is interpreted as a pilot requirement to jobs and should not be overriden. CHANGE: (#4027) Improve scalability of HTCondorCE jobs *Accounting CHANGE: (#4033) accounting clients use DIRAC.Core.Base.Client as base *DataManagementSystem FIX: (#4042) add exit handler for stored procedure FIX: (#4048) correct the header of the CSV file generated by dirac-dms-protocol-matrix *TransformationSystem FIX: (#4038) TransformationCleaningAgent cancels the Request instead of removing them *Resources CHANGE: (#4048) SE: give preference to native plugins when generating third party URLS [v6r21p4] WorkloadManagementSystem CHANGE: (#4008) Modification of utility function PilotCStoJSONSynchronizer. The modification allows to add information to created json file about the DNs fields of users belonging to 'lhcb_pilot' group. This information is needed for the second level authorization used in the Pilot Logger architecture. Also, some basic unit tests are added. *Docs CHANGE: (#4028) update instructions to install and setup runit *TransformationSystem FIX: (#4022) when a site was requested inside the job workflow description, and BulkSubmission was used, such site was not considered. *Resources FIX: (#4006) Resources/MessageQueue: add a dedicated listener ReconnectListener [v6r21p3] *Core FIX: (#4005) getDiracModules is removed, class member is used instead. FIX: (#4013) Use getCAsLocation in order to avoid non-exist os.environ['X509_CERT_DIR'] *ConfigurationSystem FIX: (#4004) BDII2CSAgent: fix for CEs with incomplete BDII info *WorkloadManagementSystem NEW: (#4016) JobAgent - added possibility to try out several CE descriptions when getting jobs in one cycle NEW: (#4016) Matcher - MultiProcessor tag is added to the resource description if appropriate NEW: (#4016) JobScheduling - MultiProcessor tag is added to the job description if it specifies multiple processor requirements FIX: (#4018) JobMonitoring.getJobParameter cast to int NEW: (#4019) added WMSAdministratorClient module, and using it throughout the code *Resources/MessageQueue CHANGE: (#4007) change the way of defining identifier format for MQ resources: accepted values are 'Topics' or 'Queues'. *DataManagementSystem CHANGE: (#4017) DIP handler internally uses bytes instead of MB NEW: (#4010) add dirac-dms-protocol-matrix script CHANGE: (#4010) remove dirac-dms-add-files script *Resources NEW: (#4016) PoolComputingElement - getDescription returns a list of descriptions with different requirements to jobs to be matched CHANGE: (#4017) Standardize sizes returned by StoragePlugins in Bytes CHANGE: (#4011) MQ: randomzied the broker list when putting message [v6r21p2] *Core CHANGE: (#3992) dirac-install does not define REQUESTS_CA_BUNDLE in the bashrc anymore NEW: (#3998) dirac-install if DIRACOS already installed and DIRACOS is not requested, it will force to install it CHANGE: (#3992) specify the ca location when calling requests CHANGE: (#3991) MySQL class prints only debug logs FIX: (#4003) dirac-install - if the DIRACOS version is not given then use the proper release version *WorkloadManagementSystem CHANGE: (#3992) specify the ca location when calling requests FIX: (#4002) Local protocols are retrieved as a list in InputDataResolution *Interfaces FIX: (#4000) Dirac.py - bug fixed: return value of getJobParameters changed that should be taken into account by the clients [v6r21p1] *WorkloadManagementSystem CHANGE: (#3989) JobDB.py - do not add default SubmitPool parameter to a job description FIX: (#3989) dirac-admin-get-site-mask - show only sites in Active state *DataManagementSystem CHANGE: (#3985) FTS3DB: getActiveJobs, those jobs are now selected that have been monitored the longest time ago. Ensure better cycling through FTS Jobs FIX: (#3987) check missing file with another string [v6r21] *Core NEW: (#3921) DictCache - allow threadLocal cache FIX: (#3936) DictCache - Fix exception upon delete FIX: (#3922) allow Script.py to accommodate specific test calls with pytest CHANGE: (#3940) dirac-install - instrument to support DiracOS FIX: (#3945) set DIRACOS environment variable before souring diracosrc CHANGE: (#3949) Removed unattended dirac-install-client.py CHANGE: (#3950) File.py - do not follow links when getting files list or size in directory via getGlobbedFiles and getGlobbedTotalSize CHANGE: (#3969) Use EOS for installing DIRAC software *FrameworkSystem FIX: (#3968) removed the old logging *ResourceStatusSystem FIX: (#3921) fix logic of the RSSCache leading in expired keys *Accounting CHANGE: (#3933) Change the columns size of the FinalMinorStatus *WorkloadManagementSystem CHANGE: (#3923) Clean PYTHONPATH from *rc when installing DIRAC from the pilot NEW: (#3941) JobDB: getJobParameters work also with list on job IDs CHANGE: (#3941) JobCleaningAgent queries for job parameters in bulk CHANGE: (#3941) Optimizers only set optimizers parameters (backported from v7r0) CHANGE: (#3970) streamlining code in OptimizerModule. Also pep8 formatting (ignore white spaces for reviewing) FIX: (#3976) fixed Banned Sites matching in TaskQueueDB FIX: (#3970) when an optimizer agent was instantiating JobDB (via the base class) and the machine was overloaded, the connection to the DB failed but this was not noticed and the agent was not working until restarted after max cycles. Now testing JobDB is valid in OptimizerModule base class and exit if not valid. *TransformationSystem CHANGE: (#3946) Remove directory listing from ValidateOutputDataAgent CHANGE: (#3946) Remove directory listing from TransformationCleaningAgent FIX: (#3967) TransformationCleaningAgent: don't return error if log directory does not exist *Interfaces CHANGE: (#3947) removed old methods going through old RMS CHANGE: (#3960) Dirac.py - getLFNMetadata returns result for both file and directory LFNs FIX: (#3966) Dirac: replace the use of deprecated function status by getJobStatus *DataManagementSystem FIX: (#3922) Fixes FTS3 duplicate transfers FIX: (#3982) respect the source limitation when picking source for an FTS transfer *MonitoringSystem CHANGE: (#3956) Change the bucket size from week to day. *Resources CHANGE: (#3933) When crating a consumer or producer then the error message must be handled by the caller. CHANGE: (#3937) MessageQueue log backends is now set to VERBOSE instead of DEBUG NEW: (#3943) SSHComputingElement - added Preamble option to define a command to be executed right before the batch system job submission command NEW: (#3953) Added the possibility to add filters to log backends to refine the output shown/stored NEW: (#3953) Resources.LogFilters.ModuleFilter: Filter that allows one to set the LogLevel for individual modules NEW: (#3953) Resources.LogFilter.PatternFilter: Filter to select or reject log output based on words FIX: (#3959) PoolComputingElement - bug fix: initialize process pool if not yet done in getCEStatus() *test CHANGE: (#3948) integration tests run with unittest now exit with exit code != 0 if failed *docs NEW: (#3974) Added HowTo section to the User Guide [v6r20p28] *WorkloadManagementSystem FIX: (#4092) pilotTools - Ensure maxNumberOfProcessors is an int [v6r20p27] *WMS FIX: (#4020) SiteDirector - do not use keywords in addPilotTQReference/setPilotStatus calls [v6r20p26] *WorkloadManagementSystem FIX: (#3932) MutiProcessorSiteDirector: get platform is checkPLatform flag is true *DataManagementSystem FIX: (#3928) `FileCatalogClient` now properly forwards function docstrings through `checkCatalogArguments` decorator, fixes #3927 CHANGE: (#3928) `Resources.Catalog.Utilities`: use functool_wraps in `checkCatalogArguments` *TransformationSystem CHANGE: (#3934) make the recursive removal of the log directory explicit in the TransformationCleaningAgent [v6r20p25] *Core FIX: (#3909) DISET - always close the socket even in case of exception *FrameworkSystem FIX: (#3913) NotificationHandler - bugfixed: changed SMTPServer to SMTP FIX: (#3914) add extjs6 support to the web compiler *docs NEW: (#3910) Added documentation on MultiProcessor jobs *WorkloadManagementSystem FIX: (#3910) TaskQueueDB - fixed strict matching with tags, plus extended the integration test *DataManagementSystem CHANGE: (#3917) FTS3: speedup by using subqueries for the Jobs table *TransformationSystem CHANGE: (#3916) use SE.isSameSE() method *Resources NEW: (#3916) Add isSameSE method to StorageElement which works for all protocols [v6r20p24] *WorkloadManagementSystem FIX: (#3904) SiteDirector fixed case with TQs for 'ANY' site [v6r20p23] *TransformationSystem NEW: (#3903) do not remove archive SEs when looking at closerSE *CORE NEW: (#3902) When the environment variable DIRAC_DEPRECATED_FAIL is set to a non-empty value, the use of deprecated functions will raise a NotImplementError exception *ConfigurationSystem FIX: (#3903) ServiceInterface - fix exception when removing dead slave *FrameworkSystem FIX: (#3901) NotificationClient - bug fix [v6r20p22] *Core FIX: (#3897) ObjectLoader returns DErrno code FIX: (#3895) more debug messages in BaseClient *ResourceStatusSystem FIX: (#3895) fixed bug in dirac-rss-set-token script *WorkloadManagementSystem FIX: (#3897) SiteDirector: using checkPlatform flag everwhere needed CHANGE: (#3894) Using JobStateUpdateClient instead of RPCClient to it CHANGE: (#3894) Using JobManagerClient instead of RPCClient to it [v6r20p20] *Core CHANGE: (#3885) Script.parseCommandLine: the called script is not necessarily the first in sys.argv *ConfigurationSystem CHANGE: (#3887) /Client/Helpers/Registry.py: Added search dirac user for ID and CA *MonitoringSystem FIX: (#3888) mqProducer field in MonitoringReporter can be set to None, and the comparison was broken. It is fixed. Also some additional checks are added. *WorkloadManagementSystem CHANGE: (#3889) removed confusing Job parameter LocalBatchID CHANGE: (#3854) TQ matching (TaskQueueDB.py): when "ANY" is specified, don't exclude task queues (fix with "Platforms" matching in mind) CHANGE: (#3854) SiteDirector: split method getPlatforms, for extension purposes *DataManagementSystem FIX: (#3884) restore correct default value for the SEPrefix in the FileCatalogClient FIX: (#3886) FTS3: remove the hardcoded srm protocol for registration FIX: (#3886) FTS3: return an empty spacetoken if SRM is not available *TransformationSystem CHANGE: (#3891) ReplicationTransformation.createDataTransformation: returns S_OK with the transformation object when it was successfully added, instead of S_OK(None) *Resources NEW: (#3886) SE - return a standard error in case the requested protocol is not available [v6r20p18] *DataManagementSystem CHANGE: (#3882) script for allow/ban SEs now accepting -a/--All switch, for allo status types *Core FIX: (#3882) ClassAdLight - fix to avoid returning a list with empty string *Resources FIX: (#3882) Add site name configuration for the dirac installation inside singularity CE *test FIX: (#3882) fully activating RSS in Jenkins tests [v6r20p17] *Core CHANGE: (#3874) dirac-create-distribution-tarball - add tests directory to the tar file and fix pylint warnings. FIX: (#3876) Add function "discoverInterfaces" again which is still needed for VMDIRAC *ConfigurationSystem CHANGE: (#3875) Resources - allow to pass a list of platforms to getDIRACPlatform() *WorkloadManagement CHANGE: (#3867) SandboxStoreClient - Returning file location in output of getOutputSandbox CHANGE: (#3875) JobDB - allow to define a list of Platforms in a job description JDL *ResourceStatusSystem CHANGE: (#3863) deprecated CSHelpers.getSites() function *Interfaces NEW: (#3872) Add protocol option to dirac-dms-lfn-accessURL CHANGE: (#3864) marked deprecated some API functions (perfect replace exists already, as specified) *Resources FIX: (#3868) GFAL2_SRM2Storage: only set SPACETOKENDESC when SpaceToken is not an empty string *Test CHANGE: (#3863) Enable RSS in Jenkins *DataManagementSystem FIX: (#3859) FTS3: resubmit files in status Canceled on the FTS server NEW: (#3871) FTS submissions can use any third party protocol NEW: (#3871) Storage plugin for Echo (gsiftp+root) FIX: (#3871) replace deprecated calls to the gfal2 API NEW: (#3871) Generic implementation for retrieving space occupancy on storage *TransformationSystem FIX: (#3865) fixed submission of parametric jobs with InputData from WorkflowTask FIX: (#3865) better logging for parametric jobs submission *StorageManagamentSystem FIX: (#3868) Fix StageRequestAgent failures for SEs without a SpaceToken *RequestManagementSystem FIX: (#3861) tests do not re-use File objects [v6r20p16] *WorkloadManagementSystem CHANGE: (#3850) the platform discovery can be VO-specific. *Interfaces CHANGE: (#3856) setParameterSequence always return S_OK/S_ERROR *TransformationSystem FIX: (#3856) check for return value on Job interface and handle it *ResourceStatusSystem FIX: (#3852) site may not have any SE [v6r20p15] *Interface FIX: (#3843) Fix the sandbox download, returning the inMemory default. *WorkloadManagementSystem FIX: (#3845) late creation on RPC in JobMonitoringClient and PilotsLoggingClient *DataManagementSystem FIX: (#3839) Update obsolete dirac-rms-show-request command in user message displayed when running dirac-dms-replicate-and-register-request *FrameworkSystem FIX: (#3845) added setServer for NotificationClient *Docs NEW: (#3847) Added some info on parametric jobs [v6r20p14] CHANGE: (#3826) emacs backup file pattern added to .gitignore *MonitoringSystem CHANGE: (#3827) The default name of the Message Queue can be changed *Core FIX: (#3832) VOMSService.py: better logging and error prevention *ConfigurationSystem FIX: (#3837) Corrected configuration location for Pilot 3 files synchronization *FrameworkSystem FIX: (#3830) InstalledComponentDB.__filterFields: fix error in "Component History Web App" when filter values are unicode *Interface CHANGE: (#3836) Dirac.py API - make the unpacking of downloaded sandboxes optional *Accounting CHANGE: (#3831) ReportGenerator: Authenticated users without JOB_SHARING will now only get plots showing their own jobs, solves #3776 *ResourceStatusSystem FIX: (#3833) Documentation update CHANGE: (#3838) For some info, use DMSHelper instead of CSHelper for better precision *RequestManagementSystem FIX: (#3829) catch more exception in the ReqClient when trying to display the associated FTS jobs [v6r20p13] *FrameworkSystem FIX: (#3822) obsolete parameter maxQueueSize in UserProfileDB initialization removed *WorkloadManagementSystem FIX: (#3824) Added Parameter "Queue" to methods invoked on batch systems by LocalComputingElement FIX: (#3818) Testing parametric jobs locally now should also work for parametric input data NEW: (#3818) Parameters from Parametric jobs are also replaced for ModuleParameters, and not only for common workflow parameters *DataManagementSystem FIX: (#3825) FileCatalogCLI: print error message when removeReplica encounters weird return value FIX: (#3819) ReplicateAndRegister: fix a problem when transferring files to multiple storage elements, if more than one attempt was needed the transfer to all SEs was not always happening. CHANGE: (#3821) FTS3Agent: set pool_size of the FTS3DB *TransformationSystem FIX: (#3820) Fix exception in TransformationCleaningAgent: "'str' object not callable" *ConfigurationSystem FIX: (#3816) The VOMS2CSAgent was not sending notification emails when the DetailedReport option was set to False, it will now send emails again when things change for a VO. CHANGE: (#3816) VOMS2CSAgent: Users to be checked for deletion are now printed sorted and line by line NEW: (#3817) dirac-admin-check-config-options script to compare options and values between the current Configuration and the ConfigTemplates. Allows one to find wrong or missing option names or just see the difference between the current settings and the default values. [v6r20p12] *Core FIX: (#3807) Glue2 will return a constant 2500 for the SI00 queue parameter, any value is needed so that the SiteDirector does not ignore the queue, fixes #3790 *ConfigurationSystem FIX: (#3797) VOMS2CSAgent: return error when VO is not set (instead of exception) FIX: (#3797) BDII2CSAgent: Fix for GLUE2URLs option in ConfigTemplate (Lower case S at the end) *DataManagementSystem FIX: (#3814) SEManager - adapt to the new meaning of the SE plugin section name FIX: (#3814) SEManager - return also VO specific prefixes for the getReplicas() and similar calls FIX: (#3814) FileCatalogClient - take into account VO specific prefixes when constructing PFNs on the fly *TransformationSystem FIX: (#3812) checking return value of jobManagerClient.getMaxParametricJobs() call [v6r20p11] *Core FIX: (#3805) ElasticSearchDB - fix a typo (itertems -> iteritems()) [v6r20p10] *Core NEW: (#3801) ElasticSearchDB - add method which allows for deletion by query NEW: (#3792) added breakDictionaryIntoChunks utility *WorkloadManagementSystem FIX: (#3796) Removed legacy "SystemConfig" and "LHCbPlatform" checks FIX: (#3803) bug fix: missing loop on pRef in SiteDirector NEW: (#3792) JobManager exposes a call to get the maxParametricJobs *TransformationSystem NEW: (#3804) new option for dirac-transformation-replication scrip `--GroupName/-R` FIX: (#3804) The TransformationGroup is now properly set for transformation created with dirac-transformation-replication, previously a transformation parameter Group was created instead. FIX: (#3792) Adding JobType as parameter to parametric jobs FIX: (#3792) WorkflowTaskAgent is submitting a chunk of tasks not exceeding the MaxParametricJobs accepted by JobManager [v6r20p9] *Core FIX: (#3794) Fix executeWithUserProxy when called with proxyUserDN, fixes exception in WMSAdministrator getPilotLoggingInfo and TransformationCleaningAgent *DataManagementSystem CHANGE: (#3793) reuse of the ThreadPool in the FTS3Agent in order to optimize the Context use *WorkloadManagementSystem FIX: (#3787) Better and simpler code and test for SiteDirector FIX: (#3791) Fix exception in TaskQueueDB.getActiveTaskQueues, triggered by dirac-admin-show-task-queues [v6r20p8] *ResourceStatusSystem FIX: (#3782) try/except for OperationalError for sqlite (EmailAction) *Core FIX: (#3785) Adjust voms-proxy-init timeouts NEW: (#3773) New Core.Utilities.Proxy.UserProxy class to be used as a contextManager FIX: (#3773) Fix race condition in Core.Utilities.Proxy.executeWithUserProxy, the $X509_USER_PROXY environment variable from one thread could leak to another, fixes #3764 *ConfigurationSystem NEW: (#3784) Bdii2CSAgent: New option **SelectedSites**, if any sites are set, only those will be updated NEW: (#3788) for CS/Registry section: added possibility to define a QuarantineGroup per VO *WorkloadManagementSystem FIX: (#3786) StalledJobAgent: fix "Proxy not found" error when sending kill command to stalled job, fixes #3783 FIX: (#3773) The solution for bug #3764 fixes a problem with the JobScheduling executor, where files could end up in the checking state with the error "Couldn't get storage metadata of some files" FIX: (#3779) Add setting of X509_USER_PROXY in pilot wrapper script, which is needed to establish pilot env in work nodes of Cluster sites. *DataManagementSystem FIX: (#3778) Added template for RegisterReplica FIX: (#3772) add a protection against race condition between RMS and FTS3 FIX: (#3774) Fix FTS3 multi-VO support by setting VO name in SE constructor. *TransformationSystem FIX: (#3789) better tests for TS agents *StorageManagamentSystem FIX: (#3773) Fix setting of the user proxy for StorageElement.getFileMetadata calls, fixes #3764 [v6r20p7] *Core FIX: (#3768) The Glue2 parsing handles some common issues more gracefully: handle cases where the execution environment just does not exist, use sensible; dummy values in this case (many sites); handle multiple execution environments at a single computing share (i.e., CERN); handle multiple execution environments with the same ID (e.g., SARA) CHANGE: (#3768) some print outs are prefixed with "SCHEMA PROBLEM", which seem to point to problems in the published information, i.e. keys pointing to non-existent entries, or non-unique IDs *Tests NEW: (#3769) allow to install DIRACOS if DIRACOSVER env variable is specified *ResourceStatusSystem CHANGE: (#3767) Added a post-processing function in InfoGetter, for handling special case of FreeDisk policies *WorkloadManagementSystem FIX: (#3767) corrected inconsistent option name for pilotFileServer CS option *TransformationSystem CHANGE: (#3766) TransformationCleaningAgent can now run without a shifterProxy, it uses the author of the transformation for the cleanup actions instead. CHANGE: (#3766) TransformationCleaningAgent: the default value for shifterProxy was removed FIX: (#3766) TaskManagerAgent: RequestTasks/WorkflowTasks: value for useCertficates to `False` instead of `None`. Fixes the broken submission when using a shifterProxy for the TaskManagerAgents [v6r20p6] *Tests CHANGE: (#3757) generate self signed certificate TLS compliant *Interfaces FIX: (#3754) classmethods should not have self! (Dirac.py) *WorkloadManagementSystem FIX: (#3755) JobManager - bug fix in __deleteJob resulting in exceptions *DataManagementSystem NEW: (#3736) FTS3 add kicking of stuck jobs FIX: (#3736) FTS3 update files in sequence to avoid mysql deadlock CHANGE: (#3736) Canceled is not a final state for FTS3 Files CHANGE: (#3736) FTS3Operations are finalized if the Request is in a final state (instead of Scheduled) FIX: (#3724) change the ps_delete_files and ps_delete_replicas_from_file_ids to not lock on MySQL 5.7 *TransformationSystem CHANGE: (#3758) re-written a large test as pytest (much less verbosity, plan to extend it) FIX: (#3758) added BulkSubmission option in documentation for WorkflowTaskAgent *RequestManagementSystem FIX: (#3759) dirac-rms-request: silence a warning, when not using the old FTS Services *ResourceStatusSystem FIX: (#3753) - style changes [v6r20p5] *Docs FIX: (#3747) fix many warnings FIX: (#3735) GetReleaseNotes.py no longer depends on curl, but the python requests packe FIX: (#3740) Fix fake environments for sqlalchemy.ext import, some code documentation pages were not build, e.g. FTS3Agent NEW: (#3762) Add --repo option, e.g. --repo DiracGrid/DiracOS, or just --repo DiracOS, fixes DIRACGrid/DIRACOS#30 *TransformationSystem FIX: (#3726) If the result can not be evaluated, it can be converted to list FIX: (#3723) TaskManagerAgentBase - add option ShifterCredentials to set the credentials to use for all submissions, this is single VO only FIX: (#3723) WorkflowTasks/RequestTasks: pass ownerDN and ownerGroup parameter to all the submission clients if using shifterProxy ownerDN and ownerGroup are None thus reproducing the original behaviour FIX: (#3723) TaskManagerAgentBase - refactor adding operations for transformation to separate function to ensure presence of Owner/DN/Group in dict entries RequestTaskAgent no longer sets shifterProxy by default. *Resources CHANGE: (#3745) Add the deprecated decorator to native XROOT plugin [v6r20p4] *DMS FIX: (#3727) use proxy location in the SECache *RMS FIX: (#3727) use downloadVOMSProxyToFile in RequestTask *TS FIX: (#3720) TaskManager - pass output data arguments as lists rather than strings to the parametric job description Docs: FIX: (#3725) AdministratorGuide TransformationSystem spell check and added a few phrases, notably for bulk submission working in v6r20p3 [v6r20p3] *Framework FIX: SystemAdministrator - Get the correct cpu usage data for each component *TS NEW: new command dirac-transformation-replication to create replication transformation to copy files from some SEs to other SEs, resolves #3700 *RMS FIX: fix integration tests to work with privileged and non privileged proxies *RSS FIX: Fix for downtime publisher: wrong column names. Avoiding dates (not reflected in web app) [v6r20p2] *Core CHANGE: (#3713) Fixes the infamous "invalid action proposal" by speeding up the handshake and not looking up the user/group in the baseStub *RequestManagementSystem CHANGE: (#3713) FowardDISET uses the owner/group of Request to execute the stub CHANGE: (#3713) owner/group of the Requests are evaluated/authorized on the server side CHANGE: (#3713) LimitedDelegation or FullDelegation are required to set requests on behalf of others -> pilot user and hosts should must them (which should already be the case) *docs NEW: (#3699) documentation on Workflow CHANGE: (#3699) update on documentation for integration tests *ConfigurationSystem CHANGE: (#3699) for pilotCS2JSONSynchronizer: if pilotFileServer is not set, still print out the content *WorkloadManagementSystem CHANGE: (#3693) introduce options for sites to choose usage of Singularity *TransformationSystem FIX: (#3706) TaskManger with bulksubmission might have occasional exception, depending on order of entries in a dictionary FIX: (#3709) TaskManager - fix the generated JobName to be of the form ProdID_TaskID FIX: (#3709) TaskManager - check the JOB_ID and PRODUCTION_ID parameters are defined in the workflow *Interfaces FIX: (#3709) Job API - do not merge workflow non-JDL parameters with the sequence parameters of the same name [v6r20p1] *WorkloadManagementSystem FIX: (#3697) Ensure retrieveTaskQueues doesn't return anything when given an empty list of TQ IDs. FIX: (#3698) Call optimizer fast-path for non-bulk jobs [v6r20] *Core NEW: MJF utility added, providing a general interface to Machine/Job Features values. NEW: DEncode - added unit tests NEW: JEncode for json based serialization NEW: Add conditional printout of the traceback when serializing/deserializing non json compatible object in DEncode (enabled with DIRAC_DEBUG_DENCODE_CALLSTACK environment variable) NEW: File.py - utility to convert file sizes between different unit NEW: new flag in dirac-install script to install DIRAC-OS on demand CHANGE: Removed deprecated option "ExtraModules" (dirac-configure, dirac-install scripts) CHANGE: dirac-deploy-scripts, dirac-install - allow command modules with underscores in their names in order for better support for the code checking tools CHANGE: dirac-distribution and related scripts - compile web code while release generation CHANGE: dirac-external-requirements - reimplemented to use preinstalled pip command rather than the pip python API FIX: dirac-distribution - fixed wrong indentation NEW: new command name for voms proxy FIX: dirac-install default behaviour preserved even with diracos options New: Add additional check in MJF utility to look for a shutdown file located at '/var/run/shutdown_time' FIX: The hardcoded rule was not taken into account when the query was coming from the web server CHANGE: VOMSService - reimplemented using VOMS REST interface FIX: MJF utility won't throw exceptions when MJF is not fully deployed at a site *Framework NEW: WebAppCompiler methods is implemented, which is used to compile the web framework NEW: add JsonFormatter for logs NEW: add default configuration to CS: only TrustedHost can upload file CHANGE: ComponentInstaller - remove the old web portal configuration data used during the installation CHANGE: MessageQueue log handler uses JsonFormatter *Monitoring CHANGE: fixes for testing in Jenkins with locally-deployed ElasticSearch FIX: fixes in the query results interpretation *Configuration FIX: ConfigurationHandler, PilotCStoJSONSynchronizer - fixes for enabling pilotCStoJSONSynchronizer, and doc NEW: dirac-admin-voms-sync - command line for VOMS to CS synchronization NEW: VOMS2CSSynchronizer - new class encapsulating VOMS to CS synchronization CHANGE: VOMS2CSAgent - reimplemented to use VOMS2CSSynchronizer *WorkloadManagementSystem NEW: StopSigRegex, StopSigStartSeconds, StopSigFinishSeconds, StopSigNumber added to JDL, which cause Watchdog to send a signal StopSigNumber to payload processes matching StopSigRegex when there are less than StopSigFinishSeconds of wall clock remaining according to MJF. NEW: PilotLoggingDB, Service and Client for handling extended pilot logging NEW: added a new synchronizer for Pilot3: sync of subset of CS info to JSON file, and sync of pilot3 files NEW: dirac-admin-get-pilotslogging script for viewing PilotsLogging NEW: Bulk job submission with protection of the operation transaction NEW: WMSHistoryCorrector and MonitoringHistoryCorrector classes inheriting from a common BaseHistoryCorrector class CHANGE: SiteDirector - refactored Site Director for better extensibility CHANGE: dirac-wms-cpu-normalization uses the abstracted DB12 benchmark script used by the HEPiX Benchmarking Working Group, and the new MJF utility to obtain values from the system and to save them into the DIRAC LocalSite configuration. CHANGE: Removed TaskQueueDirector and the other old style (WMS) *PilotDirector CHANGE: TaskQueueDB - removed PilotsRequirements table CHANGE: TaskQueueDB - added FOREIGN KEYS CHANGE: Removed gLite pilot related WMS code FIX: always initialize gPilotAgentsDB object FIX: JobManager - Added some debug message when deleting jobs FIX: Job.py - fixing finding XML file NEW: SiteDirector - added flag for sending pilot3 files CHANGE: SiteDirector - changed the way we create the pilotWrapper (better extensibility) NEW: SiteDirector - added possibility for deploying environment variables in the pilot wrapper *Workflow CHANGE: Script.py: created _exitWithError method for extension possibilities *TS FIX: TranformationCleaningAgent - just few simplifications *DMS NEW: FTS3Agent working only with the FTS3 service to replace the existing one NEW: FTS3Utilities - use correct FTS Server Selection Policy NEW: StorageElement service - getFreeDiskSpace() and getTotalDiskSpace() take into account MAX_STORAGE_SIZE parameter value CHANGE: Adding vo name argument for StorageElement CHANGE: Fixing rss to fetch fts3 server status NEW: Add a feature to the DFC LHCbManager to dump the content of an SE as a CSV file FIX: FTS3DB: sqlalchemy filter statements with "is None" do not work and result in no lines being selected NEW: FTS3Agent and FTS3DB: add functionality to kick stuck requests and delete old requests NEW: FTS3Agent - add accounting report *RMS FIX: Really exit the RequestExecutingAgent when the result queue is buggy *RSS CHANGE: Using StorageElement.getOccupancy() FIX: Initialize RPC to WMSAdministrator only once FIX: Using MB as default for the size FIX: flagged some commands that for the moment are unusable FIX: fixed documentation of how to develop commands *Resources NEW: New SingularityComputingElement to submit jobs to a Singularity container NEW: Added StorageElement.getOccupancy() method for DIP and GFAL2_SMR2 SE types CHANGE: enable Stomp logging only if DIRAC_DEBUG_STOMP environment variable is set to any value *Interfaces CHANGE: Dirac.py - saving output of jobs run with 'runLocal' when they fail (for DEBUG purposes) *Docs CHANGE: WebApp release procedure FIX: Update of the FTS3 docs *Tests FIX: add MonitoringDB to the configuration FIX: Installing elasticSeach locally in Jenkins, with ComponentInstaller support. [v6r19p25] *TransformationSystem FIX: (#3742) TransformationDB - when adding files to transformations with a multi-threaded agent, it might happen that 2 threads are adding the same file at the same time. The LFN was not unique in the DataFiles table, which was a mistake... This fix assumes the LFN is unique, i.e. if not the table had been cleaned and the table updated to be unique. [v6r19p24] *WMS FIX: (#3739) pilotTools - added --tag and --requiredTag options FIX: (#3739) pilotCommands - make NumberOfProcessors = 1 if nowhere defined (default) *Resources FIX: (#3739) CREAMComputingElement - possibility to defined CEQueueName to be used in the pilot submission command [v6r19p23] *TS FIX: (#3734) catch correct exception for ast.literal_eval [v6r19p22] *Core CHANGE: Backport from v6r20 - fixes the infamous "invalid action proposal" by speeding up the handshake and not looking up the user/group in the baseStub RMS: CHANGE: Backport from v6r20 - FowardDISET uses the owner/group of Request to execute the stub CHANGE: Backport from v6r20 - owner/group of the Requests are evaluated/authorized on the server side CHANGE: Backport from v6r20 - LimitedDelegation or FullDelegation are required to set requests on behalf of others -> pilot user and hosts should must them (which should already be the case) *API NEW: Dirac.py - running jobs locally now also works for parametric jobs. Only the first sequence will be run FIX: Dirac.py - running jobs locally will now properly work with LFNs in the inputSanbdox *DMS FIX: DMSHelpers - in getLocalSiteForSE() return None as LocalSite if an SE is at no site [v6r19p21] *Configuration FIX: Bdii2CSAgent - make the GLUE2 information gathering less verbose; Silently ignore StorageShares *Test CHANGE: backported some of the CI tools from the integration branch [v6r19p20] *StorageManagement FIX: StorageManagementDB - fixed buggy group by with MySQL 5.7 [v6r19p19] *Configuration NEW: BDII2CSAgent - new options: GLUE2URLs, if set this is queried in addition to the other BDII; GLUE2Only to turn off looking on the old schema, if true only the main BDII URL is queried; Host to set the BDII host to search NEW: dirac-admin-add-resources new option G/glue2 , enable looking at GLUE2 Schema, H/host to set the host URL to something else [v6r19p18] *Configuration CHANGE: Better logging of the Configuration file write exception *RSS FIX: SummarizeLogsAgent - fix the case when no previous history [v6r19p17] *Framework FIX: ProxyManager - if an extension has a ProxyDB, use it *RSS FIX: CSHelpers.py minor fixes [v6r19p16] *WMS FIX: pilotCommands - cast maxNumOfProcs to an int. CHANGE: pilotTools - change maxNumOfProcs short option from -P to -m. [v6r19p15] *Framework NEW: ProxyDB - allow FROM address to be set for proxy expiry e-mails *DMS CHANGE: FTSJob - FailedSize is now BIGINT in FTSJob CHANGE: FTSJob - increase the bringonline time *WMS FIX: SiteDirector won't set CPUTime of the pilot FIX: convert MaxRAM inside the pilots to int *RSS FIX: SummarizeLogsAgent: comparison bug fix FIX: Fixed sites synchronizer [v6r19p14] *WMS NEW: pilotCommands/Tools - added possibility to specify a maxNumberOfProcessors parameter for pilots CHANGE: MultiProcessorSiteDirector - allow kwargs to SiteDirector getExecutable & _getPilotOptions functions *RMS FIX: Fix a bug in ReplicateAndRegister Operation preventing files having failed once to be retried *DMS FIX: FileCatalogWithFkAndPsDB.sql - Fixes for the DFC to be compatible with strict group by mode (https://dev.mysql.com/doc/refman/5.7/en/sql-mode.html#sqlmode_only_full_group_by) *docs CHANGE: added little documentation for lcgBundles [v6r19p13] *WMS FIX: JobWrapper - added a debug message FIX: Allow non-processor related tags to match TQ in MultiProcessorSiteDirector. *Test CHANGE: improve Gfal2 integration tests by checking the metadata [v6r19p12] *Core CHANGE: QualityMapGraph - change the color map of the Quality plots *Framework FIX: Logging - remove the space after log messages if no variable message is printed, fixes #3587 *MonitoringSystem CHANGE: ElasticSearch 6 does not support multiple types, only one type is created instead. *RSS FIX: GOCDBClient - encode in utf-8, update goc db web api URL FIX: fixed bug in creation of history of status (avoid repetition of entries) *DMS FIX: fixed bug in FTSAgent initialization *WMS FIX: fix bug in dirac-wms-job-select: treating the case of jobGroup(s) not requested [v6r19p11] *Framework: CHANGE: moved column "Instance" of InstalledComponentsDB.InstalledComponent table from 64 to 32 characters *WMS FIX: JobWrapperTemplate - fix exception handling CHANGE: dirac-wms-select-jobs - new option to limit the number of selected jobs CHANGE: returning an error when sandboxes can't be unassigned from jobs (JobCleaningAgent) *RMS FIX: RequestDB - add missing JOIN in the web summary query NEW: dirac-rms-request - add option to allow resetting the NotBefore member even for non-failed requests *DMS FIX: FTSAgent - change data member names from uppercase to lower case *Interfaces CHANGE: autopep8 on the API/Dirac module *docs: NEW: added some doc about shifterProxy [v6r19p10] *Core FIX: MySQL - catch exception when closing closed connection *TS CHANGE: add possibility to get extension-specific tasks and files statuses in TransformationMonitor web application *RMS NEW: dirac-rms-request - add option --ListJobs to list the jobs for a set of requests *Resources FIX: Use parameters given at construction for SRM2 protocols List *StorageManagement FIX: use StorageElement object to get disk cache size *DMS FIX: DMSHelpers - fix case when no site is found for an SE FIX: ReplicateAndRegister - don't try and get SE metadata is replica is inactive [v6r19p9] *WMS CHANGE: DownloadInputData was instantiating all local SEs which is not necessary... Only instantiate those that are needed CHANGE: JobWrapper - use resolveSEGroup in order to allow defining SE groups including other SE groups FIX: JobDB - fixed typo in getSiteMaskStatus() method FIX: Fix getSiteMaskStatus in SiteDirector and MultiProcessSiteDirector CHANGE: WatchdogLinux - using python modules in instead of shell calls *DMS FIX: in DMSHelpers don't complain if an SE is at 0 sites *Interfaces CHANGE: Job.py - using the deprecated decorator for 2 deprecated methods *RSS FIX: EmailAction considers also CEs, not only SEs *Resources FIX: removed a useless/broken method in Resources helper FIX: marked as obsoleted two methods in Resources helper (FTS2 related) [v6r19p8] *Configuration FIX; Resources - don't overwrite queue tags if requiredtags are set. *Framework CHANGE: dirac-proxy-init - increase dirac-proxy-init CRL update frequency *Accounting CHANGE: AccountingDB - if the bucket length is part of the selected conditions, add to the grouping *WorkloadManagement FIX: ConfigTemplate.cfg - allow user access to getSiteMaskStatus *DataManagementSystem FIX: DMSHelpers - recursive resolution of SEGroup was keeping the SEGroup in the list *RSS FIX: CSHelper - getting FTS from the correct location CHANGE: use the SiteStatus object wherever possible *Resources FIX: CREAMComputingElement - added CS option for extra JDL parameters *Documentation CHANGE: point README to master and add badges for integration [v6r19p7] *WorkloadManagement FIX: SiteDirector - correct escaping in pilot template FIX: dirac-wms-get-wn-parameters - added some printouts to dirac-wms-get-wn-parameters [v6r19p6] *Core FIX: SocketInfo - log proper message on CA's init failure. *Accounting CHANGE: NetworkAgent - remove support of perfSONAR summaries and add support of raw metrics. *WMS FIX: JobDB - don't trigger exception in webSummary if a site with a single dot is in the system CHANGE: SiteDirector - added logging format and UTC timestamp to pilot wrapper FIX: JobMonitoring - fix in getJobPageSummaryWeb() for showing correct sign of life for stalled jobs *TS FIX: TransformationManager - fix for wrong method called by the Manager *RSS NEW: SiteStatus object uses the RSS Cache FIX: expiration time is a date (dirac-rss-query-db) [v6r19p5] *WMS CHANGE: ParametricJob - added getParameterVectorLength() to replace getNumberOfParameters with a more detailed check of the job JDL validity FIX: JobManagerHandler - restored the use of MaxParametricJobs configuration option *Interfaces FIX: Always use a list of LFNs for input data resolution (local run, mostly) *tests FIX: use rootPath instead of environment variable [v6r19p4] NEW: Added dummy setup.py in anticipation for standard installation procedure *Core CHANGE: SocketInfoFactory - version check of GSI at run time is removed *Configuration FIX: Resources - fix RequiredTags in getQueue() function *Interfaces FIX: fix exception when using Dirac.Job.getJobJDL *WMS FIX: SiteDirector - fix proxy validity check in updatePilotStatus, a new proxy was never created because isProxyValid returns non-empty dictionary FIX: JobMonitoring - web table was not considering correctly Failed jobs because stalled for setting the LastSignOfLife *DMS FIX: StorageFactory - avoid complaining if Access option is not in SE section CHANGE: dirac-dms-user-lfns - the wildcard flag will always assume leading "*" to match files, unless the full path was specified in the wildcard no files were previously matched *RSS FIX: CacheFeederAgent resilient to command exceptions *Resources FIX: ARCComputingElement - the proxy environment variable was assumed before the return value of the prepareProxy function was checked, which could lead to exceptions [v6r19p3] CHANGE: .pylintrc - disable redefined-variable-type CHANGE: .pylintrc - max-nested-blocks=10 due to the many tests of result['OK'] CHANGE: use autopep8 for auto-formatting with following exceptions: tabs = 2 spaces and not 4 line length check disabled (i.e. 120 characters instead of 80) Option for autopep8 are: --ignore E111,E101,E501 *Configuration FIX: retrigger the initialization of the logger and the ObjectLoader after all the CS has been loaded *WMS FIX: pilot commands will add /DIRAC/Extensions=extensions if requested FIX: SiteDirector, pilotCommands - fix support for multiple values in the RequiredTag CE parameter FIX: MultiProcessorSiteDirector - fix dictionary changed size exception *Workflow FIX: application log name can also come from step_commons.get['logFile'] *Resources CHANGE: Condor, SLURM, SSHComputingElement - added parameters to force allocation of multi-core job slots [v6r19p2] *DMS FIX: dirac-admin-allow-se: fix crash because of usage of old RSS function *RSS FIX: ResourceStatusDB - microseconds should always be 0 FIX: Multiple fixes for the RSS tests [v6r19p1] *Core FIX: ElasticSearchDB - certifi package was miscalled FIX: ElasticSearchDB - added debug messages for DB connection *Framework FIX: ComponentInstaller - handling correctly extensions of DBs found in sql files *WMS FIX: SudoComputingElement - prevent message overwriting application errors FIX: JobDB.getInputData now returns list of cleaned LFNs strings, possible "LFN:" prefix is removed *Interfaces FIX: Dirac.py - bring back treatment of files in working local submission directory [v6r19] FIX: In multiple places - use systemCall() rather than shellCall() to avoid potential shell injection problems FIX: All Databases are granting also REFERENCES grant to Dirac user to comply with more strict policies of MySQL version >= 5.7 *Accounting NEW: new functionality to plot the data gathered by perfSONARs. It allows to present jitter, one-way delay, packet-loss rate and some derived functions. FIX: compatibility of AccountingDB with MySQL 5.7 *ConfigurationSystem NEW: Allow to define FailoverURLs and to reference MainServers in the URLs *FrameworkSystem NEW: gLogger is replaced by the new logging system based on the python logging module NEW: Added ElasticSearch backend for the logging NEW: Central Backends configuration to customize their use by multiple components NEW: BundleDelivery - serves also CA's and CRL's all-in-one files NEW: added shell scripts for generating CAs and CRLs with the possibility to specify the Input and/or output directories CHANGE: can now send mails to multiple recipients using the NotificationClient CHANGE: Make the new logging system thread-safe FIX: Adapting query to MySLQ 5.7 "GROUP BY" clause FIX: TopErrorMessagesReporter - more precise selection to please stricter versions of MySQL CHANGE: ProxyGeneration - make RFC proxies by default, added -L/--legacy flag to dirac-proxy-init to force generation of no-RFC proxies *Core FIX: dirac-install - allow to use local md5 files CHANGE: X509Chain - fixes to allow robot proxies with embedded DIRAC group extension ( allow DIRAC group extension not in the first certificate chain step ) CHANGE: BaseClient - recheck the useServerCertificate while establishing connection and take it into account even if it has changed after the client object creation FIX: PlainTransport - fixed socket creation in initAsClient() NEW: Technology preview of new logging system, based on standard python logging module CHANGE: Added graphviz extension to sphinx builds FIX: Added documentation of low level RPC/DISET classes FIX: Gateway service - multiple fixes to resurrect the service and to correctly instantiate it NEW: dirac-install will change the shebang of the python scripts to use the environment python instead of the system one NEW: Security.Utilities - methods to generate all-in-one CA certificates and CRLs files NEW: ElasticSearchDB - gets CA's all-in-one file from the BundleDelivery service if needed NEW: genAllCAs.sh, genRevokedCerts.sh - DIRAC-free commands to generate all-in-one CA certificates and CRLs files CHANGE: dirac-create-distribution-tarball - removing docs and tests directories when creating release tarballs *DMS CHANGE: FTSJob - use Request wrapper for the fts3 REST interface instead of pycurl based client CHANGE: FTSHistoryView - drop FTSServer field from the view description CHANGE: FTSFile DB table: increased length of fields LFN(955), SourceSURL(1024), TargetSURL(1024) CHANGE: Uniform length of LFN to 255 across DIRAC dbs FIX: FTSJob - fix the serialization of 0 values FIX: FTSFile, FTSJob - fix SQL statement generation for stricter versions of MySQL *Resources NEW: New method in the StorageElement to generate pair of URLs for third party copy. Implement the logic to generate pair of URLs to do third party copy. This will be used mostly by FTS, but is not enabled as of now FIX: StorageElement - fix different weird behaviors in Storage Element, in particular, the inheritance of the protocol sections FIX: GFAL2 storage element: update for compatibility with GFAL2 2.13.3 APIs NEW: Introduced Resources/StorageElementBases configuration section for definitions of abstract SEs to be used in real SEs definition by inheritance *RMS NEW: dirac-rms-request - command including functionality of several other commands: dirac-rms-cancel|reset|show-request which are dropped. The required functionality is selected by the appropriate switches *RSS NEW: Put Sites, ComputingElements, FTS and Catalogs under the status control of the RSS system NEW: Rewrote RsourceStatus/ResourceManagementDB tables with sqlAlchemy (RM DB with declarative base style) NEW: SiteStatus client to interrogate site status with respect to RSS CHANGE: introduced backward compatibility of RSS services with DIRAC v6r17 clients CHANGE: moved some integration tests from pytest to unittest CHANGE: Moved ResourceStatusDB to sqlAlchemy declarative_base FIX: Automated setting of lastCheckTime and Dateffective in ResourceStatusDB and ResourceManagementDB FIX: fixes for tables inheritance and extensions FIX: fixes for Web return structure ("meta" column) FIX: ResourceStatus, RSSCacheNoThread - fixed RSS cache generation FIX: ResourceStatus - fixes for getting status from the CS information FIX: ResourceManagement/StatusDB - fixed bugs in meta parameter check FIX: fixed incompatibility between Active/InActive RSS clients return format FIX: SiteStatus - bug fixed in getSites() method - siteState argument not propagated to the service call FIX: ResourceStatus - return the same structure for status lookup in both RSS and CS cases FIX: Bug fixes in scripts getting data out of DB *Monitoring CHANGE: DBUtils - change the bucket sizes for the monitoring plots as function of the time span *WMS NEW: SiteDirector - checks the status of CEs and Sites with respect to RSS NEW: pilotCommands - new ReplaceDIRACCode command mostly for testing purposes NEW: JobAgent, JobWrapper - several fixes to allow the work with PoolComputingElement to support multiprocessor jobs NEW: JobScheduling - interpret WholeNode and NumberOfProcessors job JDL parameters and convert then to corresponding tags NEW: SiteDirector - CEs can define QueryCEFlag in the Configuration Service which can be used to disallow querying the CE status and use information from PiltAgentsDB instead NEW: The application error codes, when returned, are passed to the JobWrapper, and maybe interpreted. NEW: The JobWrapperTemplate can reschedule a job if the payload exits with status DErrno.EWMSRESC & 255 (222) FIX: SiteDirector - unlink is also to be skipped for Local Condor batch system FIX: JobDB - fixes necessary to suite MySQL 5.7 FIX: dirac-pilot, pilotTools - PYTHONPATH is cleared on pilot start, pilot option keepPP can override this FIX: WMSAdministratorHandler - make methods static appropriately FIX: Bug fix for correctly excluding WebApp extensions CHANGE: JobScheduling - more precise site name while the job is Waiting, using the set of sites at which the input files are online rather than checking Tier1s in eligible sites FIX: SiteDirector - aggregate tags for the general job availability test FIX: JobScheduling - bug fix in __sendToTQ() FIX: pilotTools,pilotCommands - pick up all the necessary settings from the site/queue configuration related to Tags and multi-processor NEW: SiteDirector - added option to force lcgBundle version in the pilot FIX: SiteDirector - if MaxWaitingJobs or MaxTotalJobs not defined for a queue, assume a default value of 10 FIX: MatcherHandler - preprocess resource description in getMatchingTaskQueues() FIX: JobDB - set CPUTime to a default value if not defined when rescheduling jobs *TS FIX: TransformationClient - fix issue #3446 for wrong file error counting in TS FIX: TransformationDB - set ExternalID before ExternalStatus in tasks BUGFIX: TransformationClient - fix a bug in the TS files state machine (comparing old status.lower() with new status) *Interfaces CHANGE: Dirac API - expose the protocol parameter of getAccessURL() CHANGE: Dirac API - added runLocal as an API method *Docs NEW: Documentation for developing with a container (includes Dockerfile) NEW: Add script to collate release notes from Pull Request comments NEW: Chapter on scaling and limitations CHANGE: Added documentation about runsv installation outside of DIRAC *tests NEW: Added client (scripts) system test CHANGE: Add to the TS system test, the test for transformations with meta-filters FIX: Minor fixes in the TS system test FIX: correctly update the DFC DB configuration in jenkins' tests [v6r17p35] *Core FIX: GOCDBClient - add EXTENSIONS & SCOPE tag support to GOCDB service queries. [v6r17p34] *SMS FIX: StorageManagerClient - fix logic for JobScheduling executor when CheckOnlyTapeSEs is its default true and the lfn is only on a tapeSE [v6r17p33] *WMS FIX: StalledJobAgent - if no PilotReference found in jobs parameters, do as if there would be no pilot information, i.e. set Stalled job Failed immediately CHANGE: DownloadInputData - job parameters report not only successful downloads but also failed ones FIX: JobDB - back port - set CPUTime to 0 if not defined at all for the given job FIX: JobDB - back port - use default CPUTime in the job description when rescheduling jobs *Resources FIX: ARCComputingElement - fix job submission issue due to timeout for newer lcg-bundles [v6r17p32] Resources: CHANGE: /Computing/BatchSystems/Condor.py: do not copy SiteDirector's shell environment variables into the job environment *WMS CHANGE: Add option to clear PYTHONPATH on pilot start [v6r17p31] *RMS FIX: ReqClient - avoid INFO message in client *WMS CHANGE: JobWrapper - allow SE-USER to be defined as another SE group (e.g. Tier1-USER) *DMS CHANGE: DMSHelpers - make resolveSEGroup recursive in order to be able to define SE groups in terms of SE groups [v6r17p30] *DMS CHANGE: StorageElement - added status(), storageElementName(), checksumType() methods returning values directly without the S_OK structure. Remove the checks of OK everywhere NEW: dirac-dms-add-file, DataManager - added option (-f) to force an overwrite of an existing file *TS: FIX: TransformationDB.py - set the ExternalID before the ExternalStatus in order to avoid inconsistent tasks if setting the ExternalID fails *StorageManagementSystem FIX: StorageManagementClient.py - return the full list of onlineSites while it was previously happy with only one *Resources FIX: HTCondorCEComputingElement.py - transfer output files(only log and err) for remote scheduler [v6r17p29] *WMS CHANGE: split time left margins in cpuMargin and wallClockMargin. Also simplified check. [v6r17p28] *WMS BUGFIX: JobScheduling - fix a bug introduced in 6r17p27 changes *Monitoring BUGFIX: MonitoringReporter - do not try to close the MQ connection if MD is not used [v6r17p27] *Configuration FIX: ConfigurationClient - allow default value to be a tuple, a dict or a set *Monitoring CHANGE: DBUtils - change bucket sizes and simplify settings *DMS FIX: DMSRequestOperationsBase, RemoveFile - allow request to not fail if an SE is temporarily banned FIX: dirac-admin-allow-se - first call of gLogger after its import *RMS CHANGE: remove scripts dirac-rms-show-request, dirac-rms-cancel-request and dirac-rms-reset-request and replace with a single script dirac-rms-request with option (default is "show") CHANGE: allow script to finalize a request if needed and set the job status appropriately *Resources FIX: LocalComputingElement - pilot jobIDs start with ssh to be compatible with pilotCommands. Still original jobIDs are passed to getJobStatus. To be reviewed *WMS CHANGE: JobScheduling - assign a job to Group.., if input files are at .. If several input replicas, assign Waiting to "MultipleInput" [v6r17p26] *Core FIX: dirac-install.py to fail when installation of lcgBundle has failed FIX: ClassAdLight - getAttributeInt() and getAttributeFloat() return None if the corresponding JDL attribute is not defined *MonitoringSystem CHANGE: The Consumer and Producer use separate connections to the MQ; If the db is not accessible, the messaged will not be consumed. *WMS FIX: JobDB - fix the case where parametric job placeholder %j is used in the JobName attribute FIX: JobDB - take into account that ClassAdLight methods return None if numerical attribute is not defined FIX: ParametricJob utility - fixed bug in evaluation of the ParameterStart|Step|Factor.X job numerical attribute [v6r17p25] *Monitoring NEW: Implemented the support of monthly indexes and the unit tests are fixed *RMS FIX: RequestExecutingAgent - fix infinite loop for duplicate requests *WMS NEW: ARCComputingElement - add support for multiprocessor jobs [v6r17p24] *WMS FIX: SiteDirector - unlink is also to be skipped for Local Condor batch system [v6r17p23] *WMS FIX: get job output for remote scheduler in the case of HTCondorCE [v6r17p22] *Framework FIX: NotificationClient - added avoidSpam flag to sendMail() method which is propagated to the corresponding service call *Integration FIX: several fixes in integration testing scripts [v6r17p21] *Core NEW: Mail.py - added mechanism to compare mail objects FIX: Grid.py - take into account the case sometimes happening to ARC CEs where ARC-CE BDII definitions have SubClusters where the name isn't set to the hostname of the machine *Framework FIX: Notification service - avoid duplicate emails mechanism [v6r17p20] *Core NEW: API.py - added __getstate__, __setstate__ to allow pickling objects inheriting API class by special treatment of internal Logger objects, fixes #3334 *Framework FIX: SystemAdministrator - sort software version directories by explicit versions in the old software cleaning logic FIX: MonitoringUtilities - sets a suitable "unknown" username when installing DIRAC from scratch, and the CS isn't initialized fully when running dirac-setup-site CHANGE: Logger - added getter methods to access internal protected variables, use these methods in various places instead of access Logger protected variables *WMS CHANGE: JobDB - removed unused CPUTime field in the Jobs table CHANGE: JobScheduling - make check for requested Platform among otherwise eligible sites for a given job, fail jobs if no site with requested Platform are available *RSS FIX: Commands - improved logging messages *SMS FIX: StorageManagerClient - instantiate StorageElement object with an explicit vo argument, fixes #3335 *Interfaces NEW: dirac-framework-self-ping command for a server to self ping using it's own certificate [v6r17p19] *Core FIX: Adler - fix checksum with less than 8 characters to be 8 chars long *Configuration FIX: VOMS2CSAgent - fix to accomodate some weird new user DNs (containing only CN field) *DMS FIX: FileCatalog - fix for the doc strings usage in file catalog CLI, fixes #3306 FIX: FileCatalog - modified recursive file parameter setting to enable usage of the index *SMS CHANGE: StorageManagerClient - try to get sites with data online if possible in getFilesToStage *RMS FIX: RequestExecutingAgent - tuning of the request caching while execution *WMS FIX: DownloadInputData - do not mistakenly use other metadata from the replica info than SEs FIX: JobScheduling - put sites holding data before others in the list of available sites FIX: JobScheduling - try and select replicas for staging at the same site as online files FIX: SiteDirector - keep the old pilot status if the new one can not be obtained in updatePilotStatus() *Resources FIX: CREAMComputingElement - return error when pilot output is missing in getJobOutput() *Monitoring FIX: DBUtils - change the buckets in order to support queries which require more than one year data. The maximum buckets size is 7 weeks [v6r17p18] *Framework NEW: SystemAdministrator - added possibility to remove old software installations keeping only a predefined number of the most recent ones. *DMS FIX: RemoveReplica - removing replica of a non-existing file is considered successful *SMS CHANGE: StorageManagerClient - restrict usage of executeWithUserProxy decorator to calling the SE.getFileMetadata only; added flag to check only replicas at tape SEs *WMS FIX: JobScheduling - added CS option to flag checking only replicas at tape SEs; fail jobs with input data not available in the File Catalog [v6r17p17] *DMS NEW: FTSAgent has a new CS parameter ProcessJobRequests to be able to process job requests only. This allows to run 2 FTS agents in parallel *Resources FIX: GFAL2_StorageBase - only set the space token if there is one to avoid problems with some SEs [v6r17p16] *Configuration FIX: VOMS2CSAgent - create user home directory in the catalog without recursion in the chown command *RMS FIX: RequestExecutingAgent - catch error of the cacheRequest() call FIX: ReqClient - enhanced log error message *SMS FIX: StorageManagerClient - treat the case of absent and offline files on an SE while staging *TS FIX: TaskManagerBase - process tasks in chunks of 100 in order to update faster the TS (tasks and files) *WMS FIX: JobScheduling - do not assume that all non-online files required staging [v6r17p15] *WMS CHANGE: StalledJobAgent - ignore or prolong the Stalled state period for jobs at particular sites which can be suspended, e.g. Boinc sites [v6r17p14] *Core FIX: PrettyPrint.printTable utility enhanced to allow multi-row fields and justification specification for each field value *Accounting NEW: DataStore - allow to run several instances of the service with only one which is enabled to do the bucketing *RMS NEW: new dirac-rms-list-req-cache command to list the requests in the ReqProxies services *Interfaces CHANGE: Dirac API - make several private methods visible to derived class [v6r17p13] *Core NEW: Proxy - added executeWithoutServerCertificate() decorator function *Resources FIX: CREAMComputingElement - split CREAM proxy renewal operation into smaller chunks for improved reliability [v6r17p12] *Framework FIX: SecurityFileLog - when the security logs are rotated, the buffer size is reduced to 1 MB to avoid gzip failures ( was 2 GBs ) *WMS FIX: pilotCommands - fix for interpreting DNs when saving the installation environment FIX: SandboxStoreClient - do not check/make destination directory if requested sandbox is returned InMemory *TS FIX: TransformationAgent CS option MaxFiles split in MaxFilesToProcess and MaxFilesPerTask, MaxFiles option is interpreted as MaxFilesPerTask for backward compatibility *Resources NEW: Added plug-ins for GSIFTP and HTTPS Storage protocols [v6r17p11] *Core FIX: ElasticSearchDB - set a very high number (10K) for the size of the ElasticSearch result *Monitoring FIX: MonitoringDB - et a very high number (10K) for the size of the ElasticSearch result *WMS FIX: pilotCommands - get the pilot environment from the contents of the bashrc script *DMS FIX: RemoveReplica - fix for the problem that if an error was set it was never reset FIX: SE metadata usage in several components: ConsistencyInspector, DataIntwgrityClient, FTSRequest, dirac-dms-replica-metadata, StageMonitorAgent, StageRequestAgent, StorageManagerClient, DownloadInputData, InputDataByProtocol [v6r17p10] *Core NEW: Logger - printing methods return True/False if the message was printed or not FIX: ElastocSearchDB - error messages demoted to warnings *Monitoring FIX: MonitoringReporter - create producers if the CS definitions are properly in place *TS CHANGE: TaskManagerPlugin - allow to redefine the AutoAddedSites for each job type [v6r17p9] *WMS BUGFIX: JobScheduling - bug fixed introduced in the previous patch NEW: pilotTools - introduced -o swicth for a generic CS option *SMS FIX: StorageManagerClient - fixes in the unit test *DMS FIX: FileManagerPs - in _getFileLFNs() - break a long list of LFNs into smaller chunks [v6r17p8] *Core NEW: DErrno.ENOGROUP error to denote proxies without DIRAC group extension embedded CHANGE: X509Chain - use DErrno.ENOGROUP error FIX: dirac-install, dirac-deploy-scripts - fixes to allow DIRAC client installation on recent MacOS versions with System Integrity Protection feature CHANGE: Proxy - added executionLock optional argument to executeWithUserProxy() decorator to lock while executing the function with user proxy FIX: Proxy - fix indentation in getProxy() preventing looping on the DNs *Framework FIX: ProxyDB - fix of error message check in completeDelegation() *WMS FIX: TaskQueueDB - when an empty TaskQueue is marked for deletion, it can still get matches which result in no selected jobs that produced unnecessary error messages FIX: JobScheduling executor - calls getFilesToStage() with a flag to lock while file lookup with user proxy; same for InputData executor for calling _resolveInputData() *TS FIX: FileReport - fix in setFileStatus() for setting status for multiple LFNs at once *SMS FIX: StorageManagerClient - in getFilesToStage() avoid using proxy if no files to check on a storage element *Resources FIX: GFAL2_XROOTStorage - fix to allow interactive use of xroot plugin FIX: GFAL2_StorageBase - enable IPV6 for gsiftp [v6r17p7] *DMS FIX: dirac-dms-user-lfns - do not print out empty directories *WMS FIX: InputData Executor, JobWrapper - use DataManager.getReplicasForJobs() for getting input data replicas *TS FIX: TransformationAgent - use DataManager.getReplicasForJobs() for transformations creating jobs [v6r17p6] *DMS NEW: DataManager - add key argument forJobs (default False) in getReplicas() in order to get only replicas that can be used for jobs (as defined in the CS); added getReplicasForJobs(), also used in the Dirac API *SMS FIX: Stager agents - monitor files even when there is no requestID, e.g. dCache returns None when staging a file that is already staged *Resources FIX: StorageFactory - bug fixes when interpreting SEs inheriting other SE parameters NEW: Test_StorageFactory unit test and corresponding docs FIX: Torque - some sites put advertising in the command answer that can not be parsed: redirect stderr to /dev/null [v6r17p5] *Resources FIX: LcgFileCatalogClient - do not evaluate GUID if it is not a string [v6r17p4] *Configuration FIX: Utilities - fixed interpretation of weird values of GlueCEPolicyMaxWallClockTime BDII parameter; newMaxCPUTime should is made integer *Framework FIX: Logger - make subloggers processing messages with the same level as the parent logger *Docs NEW: Updated documentation in several sections *DMS FIX: RemoveReplica operation - don't set file Done in RemoveReplicas if there is an error [v6r17p3] *RSS FIX: Synchronizer - the sync method removes the resources that are no longer in the CS from the DowntimeCache table *DMS CHANGE: dirac-dms-find-lfns - added SE switch to look for files only having replicas on a given SE (list) *TS FIX: TaskManager - optimization of the site checking while preparing job; optimized creation of the job template *Resources CHANGE: GFAL2_SRM2Storage, SRM2Storage - added gsiftp to the list of OUTPUT protocols [v6r17p2] *Monitoring FIX: ElasticSearchDB - fixes required to use host certificate for connection; fixes required to pass to version 5.0.1 of the elasticsearch.py binding [v6r17p1] *RSS FIX: GOCDBSync - make commmand more verbose and added some minor fixes [v6r17] *Core FIX: Adler - check explicitly if the checksum value is "False" FIX: install_site.sh - added command line option to choose DIRAC version to install NEW: ComponentInstaller - added configuration parameters to setup NoSQL database *Framework CHANGE: Logger - test level before processing string (i.e. mostly converting objects to strings) CHANGE: dirac-proxy-init - check and attempt to update local CRLs at the same time as generating user proxy CHANGE: ProxyManager service - always store the uploaded proxy even if the already stored one is of the same validity length to allow replacement in case of proxy type changes, e.g. RFC type proxies *DMS NEW: Next in implementation multi-protocol support for storage elements. When performing an action on the StorageElement, instead of looping over all the protocol plugins, we loop over a filtered list. This list is built taking into account which action is taken (read vs write), and is also sorted according to lists defined in the CS. The negotiation for third party transfer is also improved: it takes into account all possible protocols the source SE is able to produce, and all protocols the target is able to receive as input. NEW: StorageElement - added methods for monitoring used disk space FIX: ReplicateAndRegister - fix the case when checksum is False in the FC NEW: DMSHelpers - get list of sites from CS via methods; allow to add automatically sites with storage *RSS NEW: FreeDiskSpace - added new command which is used to get the total and the remaining disk space of all dirac storage elements that are found in the CS and inserts the results in the SpaceTokenOccupancyCache table of ResourceManagementDB database. NEW: GOCDBSync command to ensure that all the downtime dates in the DowntimeCache table are up to date Resources* NEW: Updated Message Queue interface: MQ service connection management, support for SSL connections, better code arrangement *Workflow FIX: Modulebase, Script - avoid too many unnecessarily different application states *WMS FIX: JobStateUpdate service - in setJobStatusBulk() avoid adding false information when adding an application status *TS FIX: TaskManager, TaskManagerAgentBase - standardize the logging information; removed unnecessary code; use iterators wherever possible NEW: Introduced metadata-based filters when registering new data in the TS as catalog [v6r16p6] *WMS NEW: Added MultiProcessorSiteDirector section to the ConfigTemplate.cfg *DMS FIX: FileCatalogClient - added missing read methods to the interface description getDirectoryUserMetadata(), getFileUserMetadata() [v6r16p5] FIX: included patches from v6r15p27 [v6r16p4] FIX: applied fixes from v6r15p26 [v6r16p3] FIX: incorporated fixes from v6r15p25 [v6r16p2] *Configuration CHANGE: VOMS2CSAgent - remove user DNs which are no more in VOMS. Fixes #3130 *Monitoring CHANGE: WMSHistory - added user, jobgroup and usergroup selection keys *DMS FIX: DataManager - retry checksum calculation on putAndRegister, pass checksum to the DataManager object in the FailoverTransfer object. FIX: DatasetManager, FileCatalogClientCLI - bug fixes in the dataset management and commands *WMS CHANGE: JobManager - added 'Killed' to list of jobs status that can be deleted [v6r16p1] *Monitoring CHANGE: MonitorinDB - allow to use more than one filter condition *WMS CHANGE: StalledJobAgent - send a kill signal to the job before setting it Failed. This should prevent jobs to continue running after they have been found Stalled and then Failed. [v6r16] *Core CHANGE: dirac-install, dirac-configure - use Extensions options consistently, drop ExtraModule option CHANGE: dirac-install - use insecure ssl context for downloading files with urllib2.urlopen CHANGE: GOCDBClient - replaced urllib2 with requests module FIX: dirac-setup-site - added switch to exitOnError, do not exit on error by default CHANGE: Added environment variables to rc files to enable certificates verification (necessary for python 2.7.9+) FIX: ComponentInstaller - always update CS when a database is installed, even if it is already existing in the db server FIX: SSLSocketFactory - in __checkKWArgs() use correct host address composed of 2 parts *Framework FIX: SystemAdministrator service - do not install WebAppDIRAC by default, only for the host really running the web portal *Accounting FIX: JobPolicy - remove User field from the policy conditions to fix a problem that non-authenticated user gets more privileges on the Accounting info. *Monitoring NEW: New Monitoring system is introduced to collect, analyze and display various monitoring information on DIRAC components status and behavior using ElasticSearch database. The initial implementation is to collect WMSHistory counters. *DMS NEW: MoveReplica operation for the RMS system and a corresponding dirac-dms-move-replica-request comand line tool *Resources NEW: MessageQueue resources to manage MQ connections complemented with MQListener and MQPublisher helper classes NEW: SudoComputingElement - computing element to execute payload with a sudo to a dedicated UNIX account [v6r15p27] *Configuration FIX: CSAPI - changed so that empty but existing options in the CS can be still modified [v6r15p26] *WMS FIX: SandboxStoreClient - ensure that the latest sandbox is returned in the Web portal in the case the job was reset. [v6r15p25] *Resources FIX: HTCondorCEComputingElement - cast useLocalSchedd to bool value even if it is defined as srting [v6r15p24] *Resources CHANGE: HTCondorCE - added option to use remote scheduler daemon [v6r15p23] *DMS FIX: dirac-dms-find-lfns - fixed bug causing generl script failure [v6r15p22] *Interfaces CHANGE: Dirac API - add possibility to define the VO in the API CHANGE: Dirac API - add checkSEAccess() method for checking SE status [v6r15p21] *WMS FIX: removed default LCG version from the pilot (dirac-install will use the one of the requested release) *RMS FIX: reject bad checksum [v6r15p20] *Framework FIX: SystemAdministratorHandler - in updateSoftware() put explicitly the project name into the command FIX: ComponentInstaller - added baseDir option to the mysql_install_db call while a fresh new database server installation [v6r15p19] *Core FIX: dirac-install - lcg-binding version specified in the command switch overrides the configuration option value *DMS FIX: RemoveFile operation - Remove all files that are not at banned SEs *TMS FIX: FileReport - after successful update of input files status, clear the cache dictionary to avoid double update [v6r15p18] *Configuration FIX: Utilities - take into account WallClock time limit while the MaxCPUTime evaluation in the Bdii@CSAgent *DMS FIX: FTSJob - specify checksum type at FTS request submission *StorageManagement FIX: StorageManagerClient - in getFilesToStage() avoid exception in case of no active replicas *Resources FIX: StorageBase - in getParameters() added baseURL in the list of parameters returned *WMS FIX: CPUNormalization - minor code rearrangement [v6r15p17] *Core CHANGE: GOCDBClient - catch all downtimes, independently of their scope FIX: LSFTimeLeft - accept 2 "word" output from bqueues command CHANGE: dirac-install - create bashrc/cshrc with the possibility to define installation path in the $DIRAC env variable, this is needed for the cvmfs DIRAC client installation [v6r15p16] *Core CHANGE: AgentModule - added a SIGALARM handler to set a hard timeout for each Agent cycle to avoid agents stuck forever due to some faults in the execution code *DMS FIX: DataManager - cache SE status information in filterTapeReplicas() to speed up execution *WMS BUGFIX: InputDataByProtocol - the failed resolution for local SEs was not considered correctly: if there were other SEs that were ignored (e.g. because on tape) *TS FIX: TransformationAgent - in getDataReplicasDM() no need to get replica PFNs [v6r15p15] *Configuration CHANGE: VOMS2CSAgent - added new features: deleting users no more registered in VOMS; automatic creation of home directories in the File Catalog for new users *WMS CHANGE: JobScheduling - correct handling of user specified sites in the executor, including non-existent (misspelled) site names FIX: CPUNormalization - accept if the JOBFEATURES information is zero or absent [v6r15p14] *Core FIX: BaseClient - proper error propagation to avoid excessive output in the logger *Configuration CHANGE: Resources helper - in getStorageElementOptions() dereference SEs containing BaseSE and Alias references *Accounting FIX: AccountingDB - changes to use DB index to speed-up removal query *DMS CHANGE: DMSHelpers - define SE groups SEsUsedForFailover, SEsNotToBeUsedForJobs, SEsUsedForArchive in the Operations/DataManagement and use them in the corresponding helper functions FIX: FTSJob - temporary fix for the FTS rest interface Request object until it is fixed in the FTS REST server *Resources FIX: HTCondorCEComputingElement - check that some path was found in findFile(), return with error otherwise CHANGE: ARCComputingElement - consider jobs in Hold state as Failed as they never come back CHANGE: ARCComputingElement - do not use JobSupervisor tool for bulk job cancellation as it does not seem to work, cancel jobs one by one FIX: ARCComputingElement - ensure that pilot jobs that are queued also get their proxies renewed on ARC-CE *WMS FIX: SiteDirector - ensure that a proxy of at least 3 hours is available to the updatePilotStatus function so that if it renews any proxies, it's not renewing them with a very short proxy [v6r15p13] *Resources FIX: HTCondorCEComputingElement - fixed location of log/output files *TS FIX: ValidateOutputDataAgent - works now with the DataManager shifter proxy [v6r15p12] *Core FIX: Graphs - make sure matplotlib package is always using Agg backend FIX: cshrc - added protection for cases with undefined environment variables NEW: AuthManager - added possibility to define authorization rules by VO and by user group *Configuration NEW: Resources, ComputingElement(Factory) - added possibility to define site-wide CE parameters; added possibility to define common parameters for a given CE type. *Framework FIX: SystemAdministrator service - avoid using its own client to connect to itself for storing host information FIX: SystemAdministratorClientCLI, dirac-populate-component-db - fix insertion of wrongly configured component to the ComponentMonitorDB *DMS FIX: FileCatalog service - fix the argument type for getAncestor(), getDescendents() *WMS NEW: JobCleaningAgent - add an option (disabled by default) to remove Jobs from the dirac server irrespective of their state *Resources CHANGE: HTCondorCE - added new configurable options - ExtraSubmitString, WorkingDirectory DaysToKeepLogs [v6r15p11] *Framework NEW: dirac-proxy-destroy command to destroy proxy locally and in the ProxyManager service CHANGE: ProxyManagerClient - reduce the proxy caching time to be more suitable for cases with short VOMS extensions *Configuration FIX: VOMS2CSAgent - fixed typo bug in execute() *RMS FIX: RequestTask - fix if the problem when the processing of an operation times out, there was no increment of the attempts done. *DMS FIX: FTSAgent - avoid FTS to fetch a request that was canceled *Resources FIX: HTCondorCE - protect against non-standard line in 'job status' list in the getJobStatus() CHANGE: ComputingElement - reduce the default time length of the payload proxy to accomodate the case with short VOMS extensions [v6r15p10] *Core FIX: MySQL - do not print database access password explicitly in the logs *Configuration CHANGE: VOMS2CSAgent - show in the log if there are changes ready to be committed CHANGE: Bdii2CSAgent - get information from alternative BDII's for sites not existing in central BDII *Framework FIX: ComponentInstaller - fixed location of stop_agent file in the content of t file of the runsv tool *RMS FIX: Changed default port of ReqProxy service to 9161 from 9198 *Resources FIX: BatchSystem/Condor, HYCondroCEComputingElement - more resilient parsing of the status lookup command FIX: CREAMComputingElement - in case of glite-ce-job-submit error print our both std.err and std.out for completeness and better understanding *DMS FIX: FileCatalogClient - bug fix in getDirectoryUserMetadata() *Interfaces FIX: Dirac - in replicateFile() in case of copying via the local cache check if there is another copy for the same file name is happening at the same time [v6r15p9] *Configuration FIX: fixed CS agents initialization bug *DMS FIX: fixed inconsistency between DataIntegrity and ConsistencyInspector modules *Interfaces FIX: Fix download of LFNs in InputSandbox when running job locally [v6r15p8] *Configuration NEW: Added DryRun option for CS agents (false by default, True for new installations) [v6r15p7] *Core CHANGE: Enabled attachments in the emails *TS *CHANGE: Added possibility for multiple operations in Data Operation Transformations [v6r15p6] *Resources FIX: FCConditionParser: ProxyPlugin handles the case of having no proxy *WMS FIX: MJF messages correctly parsed from the pilot NEW: Added integration test for TimeLeft utility and script calling it [v6r15p5] Included fixes from v6r14p36 patch release *Framework FIX: added GOCDB2CSAgent in template FIX: Fixed permissions for HostLogging *DMS FIX: Introduced hopefully temporary fix to circumvent globus bug in gfal2 *WMS: FIX: added test for MJF and made code more robust *RSS NEW: HTML notification Emails [v6r15p4] Included fixes from v6r14p35 patch release *Core NEW: Added a new way of doing pfnparse and pfnunparse using the standard python library. The two methods now contains a flag to know which method to use. By default, the old hand made one is used. The new one works perfectly for all standard protocols, except SRM *RSS FIX: dirac-rss-sync - command fixed to work with calling services rather than databases directly *Resources CHANGE: In multiple Storage classes use pfnparse and pfnunparse methods to manipulate url strings instead of using just string operations NEW: A new attribute is added to the storage plugins: DYNAMIC_OPTIONS. This allows to construct URLs with attributes going at the end of the URL, in the form ?key1=value1&key2=value2 This is useful for xroot and http. [v6r15p3] Included changes from v6r14p34 patch release *Accounting FIX: DataStoreClient - catch all exceptions in sending failover accounting requests as it could disrupt the logic of the caller *DMS CHANGE: dirac-dms-show-se-status - added switches to show SEs only accessible by a given VO and SEs not assigned to any VO FIX: dirac-dms-replicate-and-register-request - prints out the new request IDs to allow their monitoring by ID rather than possibly ambiguous request name [v6r15p2] *WMS FIX: pilotCommands - protect calls to external commands in case of empty or erroneous output FIX: Matcher - fixed bug in the tag matching logic: if a site presented an empty Tag list instead of no Tag field at all, it was interpreted as site accepts all the tags FIX: Matcher - matching parameters are printed out in the Matcher rather than in the TaskQueueDB, MaxRAM and Processors are not expanded into tags [v6r15p1] Included patches for v6r14p32 *Configuration CHANGE: Resources helper - remove "dips" protocol from the default list of third party protocols *Resources FIX: XROOTStorage - bug fixed in __createSingleDirectory() - proper interpretation of the xrootClient.mkdir return status FIX: XROOTStorage unit test reenabled by mocking the xrootd import [v6r15] Removed general "from DIRAC.Core.Utilities import *" in the top-level __init__.py Made service handlers systematically working with unicode string arguments Added requirements.txt and Makefile in the root of the project to support pip style installation DIRAC documentation moved to the "docs" directory if the DIRAC project from the DIRACDocs separate project. *Accounting CHANGE: INTEGER -> BIGINT for "id" in "in" accountingDB tables *Core NEW: The S_ERROR has an enhanced structure containing also the error code and the call stack from where the structure was created NEW: DErrno module to contain definitions of the DIRAC error numbers and standard descriptions to be used from now on in any error code check CHANGE: gMonitor instantiation removed from DIRAC.__init__.py to avoid problems in documentation generation CHANGE: removed Core.Utilities.List.sortList (sorted does the job) CHANGE: removed unused module Core.Utilities.TimeSeries NEW: dirac-install - makes us of the DIRAC tar files in CVMFS if available NEW: dirac-install-client - a guiding script to install the DIRAC client from A to Z CHANGE: dirac-install - when generating bashrc and cshrc scripts prepend DIRAC paths to the ones existing in the environment already NEW: MJFTimeLeft - using Machine JOb features in the TimeLeft utility FIX: BaseClient - only give warning log message "URL banned" when one of the service URLs is really banned CHANGE: DISET components - improved logic of service URL retries to speedup queries in case of problematic services NEW: dirac-rss-policy-manager - allows to interactively modify and test only the policy section of Dirac.cfg FIX: XXXTimeLeft - do not mix CPU and WallTime values FIX: ComponentInstaller - longer timeout for checking components PID (after restart) CHANGE: Proxy - in executeWithUserProxy() when multiple DNs are present, try all of them CHANGE: List utility - change uniqueElements() to be much faster NEW: Platform - added getPlatform() and getPlatformTuple() utilities to evaluate lazily the DIRAC platform only when it is needed, this accelerates DIRAC commands not needing the platform information. *Configuration NEW: GOCDB2CSAgent agent to synchronize GOCDB and CS data about perfSONAR services NEW: VOMS2CSAgent to synchronize VOMS user data with the DIRAC Registry CHANGE: ConfigurationData - lazy config data compression in getCompressedData() *Framework CHANGE: SystemAdministratorIntegrator - make initial pinging of the hosts in parallel to speed up the operation CHANGE: InstalledComponentsDB - table to cache host status information populated by a periodic task NEW: ComponentInstaller Client class to encapsulate all the installation utilities from InstallTools module NEW: SystemAdministratorClientCLI - added uninstall host command NEW: SystemAdministratorClientCLI - added show ports command NEW: SystemAdministratorHandler - added getUsedPorts() interface NEW: SystemAdministratorHandler - show host command shows also versions of the Extensions NEW: InstalledComponentsDB - added Extension field to the HostLogging table FIX: SystemLoggingDB - fixed double creation of db tables *Accounting FIX: DataStoreClient - Synchronizer based decorators have been replaced with a simple lock as they were blocking addRegister() during every commit(); *RSS NEW: CE Availability policy, closing #2373 CHANGE: Ported setStatus and setToken rpc calls to PublisherHandler from LHCb implementation NEW: E-mails generated while RSS actions are now aggregated to avoid avalanches of mails NEW: dirac-rss-sync is also synchronizing Sites now *DMS CHANGE: FileCatalogClient - make explicit methods for all service calls CHANGE: DataManager, StorageElement - move physical accounting the StorageElement CHANGE: FileCatalog - added recursive changePathXXX operations CHANGE: FileCatalog contained objects have Master attribute defined in the CS. Extra check of eligibility of the catalogs specified explicitely. No-LFN write methods return just the Master result to be compatible with the current use in the clients. CHANGE: Removed LcgFileCatalogXXX obsoleted classes NEW: ConsistencyInspector class to perform data consistency checks between different databases CHANGE: FileCatalog(Client) - refactored to allow clients declare which interface they implement NEW: FileCatalog - conditional FileCatalog instantiation based on the configured Operations criteria *TS CHANGE: TransformationDB table TaskInputs: InputVector column from BLOB to MEDIUMTEXT FIX: TaskManager - fix bug in case there is no InputData for a task, the Request created for the previous task was reassigned NEW: TaskManager - possibility to submit one bulk job for a series of tasks *WMS NEW: TaskQueueDB - possibility to present requirements in a form of tags from the site( pilot ) to the jobs to select ones with required properties FIX: JobWrapper - the InputData optimizer parameters are now DEncoded CHANGE: JobAgent - add Processors and WholeNode tags to the resources description CHANGE: SiteDirector - flag to always download pilot output is set to False by default FIX: SiteDirector - using PilotRunDirectory as WorkingDirectory, if available at the CE level in the CS. Featire requested in issue #2746 NEW: MultiProcessorSiteDirector - new director to experiment with the multiprocessor/ wholeNode queues CHANGE: JobMemory utility renamed to JobParameters CHANGE: CheckWNCapabilities pilot command changed to get WN parameters from the Machine Job Features (MJF) - NumberOfProcessors, MaxRAM NEW: JobManager, ParametricJob - utilities and support for parametric jobs with multiple parameter sequences NEW: SiteDirector - added logic to send pilots to sites with no waiting pilots even if the number of already sent pilots exceeds the number of waiting jobs. The functionality is switched on/off by the AddPilotsToEmptySites option. *RMS FIX: Request - fix for the case when one of the request is malformed, the rest of the requests could not be swiped FIX: ReqProxyHandler - don't block the ReqProxy sweeping if one of the request is buggy CHANGE: ReqProxyHandler - added monitoring counters NEW: ReqProxyHandler - added interface methods to list and show requests in a ReqProxy *Resources FIX: SRM2Storage - do not add accounting to the output structure as it is done in the container StorageElement class CHANGE: Add standard metadata in the output of all the Storage plugins *Interfaces NEW: Job API - added setParameterSequence() to add an arbitrary number of parameter sequences for parametric jobs, generate the corresponding JDL *tests NEW: The contents of the TestDIRAC package is moved into the tests directory here [v6r14p39] Patch to include WebApp version v1r6p32 [v6r14p38] *Core CHANGE: Unhashable objects as DAG graph nodes *RMS CHANGE: Added possibility of constant delay for RMS operations [v6r14p37] *Core NEW: Added soft implementation of a Direct Acyclic Graph *Configuration FIX: Bdii2CSAgent finds all CEs of a site (was finding only one) *Resources FIX: Make sure transferClient connects to the same ProxyStorage instance [v6r14p36] *Core FIX: Sending mails to multiple recipients was not working *WMS FIX: Allow staging from SEs accessible by protocol [v6r14p35] *Core FIX: SOAPFactory - fixes for import statements of suds module to work with the suds-jurko package that replaces the suds package *Resources FIX: BatchSystems.Torque - take into account that in some cases jobID includes a host name that should be stripped off FIX: SSHComputingElement - in _getJobOutputFiles() fixed bug where the output of scpCall() call was wrongly interpreted FIX: ProxyStorage - evaluate the service url as simple /DataManagement/StorageElementProxy to solve the problem with redundant StorageElementProxy services with multiple possible urls *RSS CHANGE: Configurations.py - Added DTScheduled3 policy (3 hours before downtime) *WMS FIX: pilotCommands - take into account that in the case of Torque batch system jobID includes a host name that should be stripped off [v6r14p34] *Configuration FIX: Bdii2CSAgent - reinitilize the BDII info cache at each cycle in order not to carry on obsoleted stuff. Fixes #2959 *Resources FIX: Slurm.py - use --partition rather --cluster for passing the DIRAC queue name FIX: DIPStorage - fixed bug in putFile preventing third party-like transfer from another DIPS Storage Element. Fixes #2413 *WMS CHANGE: JobWrapper - added BOINC user ID to the job parameters FIX: pilotCommands - interpret SLURM_JOBID environment if present FIX: WMSClient - strip of comments in the job JDL before any processing. Passing jdl with comments to the WMS could provoke errors in the job checking. [v6r14p33] *WMS FIX: JobAgent - included a mechanism to stop JobAgent if the host operator creates /var/lib/dirac_drain FIX: CPUNormalization - fixed a typo in getPowerFromMJF() in the name of the exception log message [v6r14p32] *Core FIX: InstallTools - getStartupComponentStatus() uses "ps -p " variant of the system call to be independent of the OS differences *DMS FIX: RemoveReplica - bulkRemoval() was modifying its input dict argument and returning it, which was useless, only modify argument *WMS CHANGE: CPUNormalization - get HS'06 worker node value from JOBFEATURES if available *RMS FIX: ReqClient - bug fixed preventing the client to contact multiple instances of ReqManager service [v6r14p31] *DMS FIX: FTSAgent - if a file was not Scheduled, the FTSAgent was setting it Done even if it had not been replicated. *Workflow FIX: FailoverRequest - forcing setting the input file Unused if it was already set Processed [v6r14p30] *Framework BUGFIX: MonitoringHandler - in deleteActivities() use retVal['Message'] if result is not OK *Resources FIX: XROOTStorage - in getFile() evaluate file URL without URL parameters in __putSingleFile() use result['Message'] in case of error *RMS FIX: dirac-rms-cancel-request - fixed crash because of gLogger object was not imported *TS FIX: TransformationCLI - in resetProcessedFile() added check that the Failed dictionary is present in the result of a call [v6r14p29] *Core FIX: Time - skip the effect of timeThis decorator if not running interractively *DMS FIX: DataManager - in getFile(), select preferentially local disk replicas, if none disk replicas, if none tape replicas FIX: DataManager - avoid changing argument of public method checkActiveReplicas() FIX: FTSAgent - wait 3 times longer for monitoring FTS jobs if Staging *Accounting CHANGE: Jobs per pilot plot is presented as Quality plot rather than a histogram *WMS CHANGE: dirac-wms-cpu-normalization - reduce memory usage by using xrange() instead of range() in the large test loop [v6r14p28] *TS FIX: TaskManager - protection against am empty task dictionary in prepareTransformationTasks() FIX: Test_Client_TransformationSystem - fixes ti run in the Travis CI environment *WMS FIX: JobMemory - use urllib instead of requests Python module as the latter can be unavailable in pilots. [v6r14p27] *Core FIX: PlainTransport,SocketInfoFactory - fix for the IPv6 "Address family not supported by protocol" problems *Interfaces NEW: Dirac.py - in ping()/pingService() allow to ping a specific URL *Resources FIX: LcgFileCatalogClient - convert LFN into str in __fullLfn to allow LFNs in a unicode encoding *WMS FIX: JobWrapper - set the job minor status to 'Failover Request Failed' if the failover request fails sending *TS FIX: TransformationDB - in getTransformationTasks(),getTaskInputVector forward error result to the callers FIX: TaskManager - in case there is no InputData for a task, the Request created for the previous task was reassigned. This fixes this bug. *tests FIX: several fixes to satisfy on-the-fly unit tests with teh Travis CI service [v6r14p26] NEW: Enabled on-the-fly tests using the Travis-CI service *Core FIX: Subprocess - fix two potential infinite loops which can result in indefinite output buffer overflow *WMS FIX: JobScheduling executor - check properly if staging is allowed, it was always True before [v6r14p25] *Core FIX: Subprocess - more detailed error log message in case ov output buffer overflow *DMS FIX: DataManager - fix for getActiveReplicas(): first check Active replicas before selecting disk SEs *Resources FIX: StorageElementCache - fixes to make this class thread safe FIX: StorageFactory - fix in getConfigStorageProtocols() to properly get options for inheriting SE definitions [v6r14p24] *Accounting FIX: Plots, JobPlotter - fix sorting by plot labels in case the enddata != "now" *DMS FIX: dirac-dms-user-lfns - add error message when proxy is expired [v6r14p23] *Interfaces FIX: Job.py - setCPUTime() method sets both CPUTime and MaxCPUTime JDL parameters for backward compatibility. Otherwise this setting was ignored by scheduling *TS BUGFIX: TaskManager - bug fixed in submitTransformationTasks in getting the TransformationID [v6r14p22] CHANGE: Multiple commands - permissions bits changed from 644 to 755 *Framework FIX: UserProfileDB - in case of desktop name belonging to two different users we have to use both desktop name and user id to identify the desktop *WMS BUGFIX: JobWrapperTemplate - bug fixed in evaluation of the job arguments *TMS CHANGE: TaskManager - added TransformationID to the log messages [v6r14p21] *DMS CHANGE: dirac-admin-allow(ban)-se - allow an SE group to be banned/allowed *SMS FIX: RequestPreparationAgent - fix crash in execute() in case no replica information available *WMS FIX: TaskQueueDB, PilotAgentsDB - escape DN strings to avoid potential SQL injection FIX: JobWrapperTemplate - pass JobArguments through a json file to fix the case of having apostrophes in the values *TMS FIX: TransformationAgent - in processTransformation() fix reduction of number of files [v6r14p20] *WMS FIX: SandboxMetadataDB - escape values in SandboxMetadataDB SQL queries to accommodate DNs containing apostrophe [v6r14p19] *Core NEW: CLI base class for all the DIRAC CLI consoles, common methods moved to the new class, XXXCLI classes updated to inherit the base class FIX: Network - fix crash when path is empty string, fixes partly #2413 *Configuration FIX: Utilities.addToChangeSet() - fix the case when comma is in the BDII Site description followed by a white space, the description string was constantly updated in the CS *Interfaces FIX: Dirac.py - in retrieveRepositorySandboxes/Data - "Retrieved" and "OutputData" key values are strings '0' in the jobDict when a repository file is read, need to cast it to int *DMS FIX: RegisterReplica - if operation fails on a file that no longer exists and has no replica at that SE, consider the operation as Done. *Resources FIX: ARCComputingElement - bug fix in getJobOutput in using the S_ERROR() [v6r14p18] *Core FIX: VOMSService - attGetUserNickname() can only return string type values FIX: dirac-deploy-scripts - install DIRAC scripts first so that they can be overwritten by versions from extensions *Framework FIX: dirac-populate-component-db - bug fixed to avoid duplicate entries in the database *TS FIX: TaskManager - do not use ReqProxy when submitting Request for Tasks, otherwise no RequestID can be obtained *Interfaces CHANGE: Dirac.py - increase verbosity of a error log message in selectJobs *Resources FIX: XROOTStorage - fixed KeyError exception while checking file existence FIX: ARCComputingElement - in getJobOutput test for existence of an already downloaded pilot log [v6r14p17] *Core FIX: Service.py - use the service name as defined in the corresponding section in the CS and not the name defined in service Module option. This fixes the problem with the StorageElement service not interpreting properly the PFN name and using a wrong local data path. *Resources CHANGE: ARCComputingElement - if the VO is not discoverable from the environment, use ARC API call in the getCEStatus, use ldapsearch otherwise [v6r14p16] *Resources CHANGE: ARC Computing Element automatically renew proxies of jobs when needed [v6r14p15] *Core FIX: VOMS.py - Fixed bug that generates proxies which are a mix between legacy and rfc proxies. *DMS CHANGE: Allow selecting disk replicas in getActiveReplicas() and getReplicas() *WMS CHANGE: Use the preferDisk option in the InputData optimizer, the TransformationAgent and in the Interface splitter [v6r14p14] *Core FIX: VOMS.py - return RFC proxy if necessary after adding the VOMS extension *Configuration FIX: Validate maxCPUTime and Site description value *Resources FIX: XROOTStorage - changes to allow third party transfers between XROOT storages CHANGE: HTCondorCEComputingElement - the Condor logging can now be obtained in the webinterface; SIGTERM (instead of SIGKILL) is send to the application in case jobs are killed by the host site; when pilots are put in held status we kill them in condor and mark them as aborted. *WMS FIX: pilotCommands - fixes for intrepreting tags in the pilot [v6r14p13] *WMS FIX: pilot commands CheckCECapabilities and CheckWNCapabilities were not considering the case of missing proxy [v6r14p12] *Core FIX: allow a renormalization of the estimated CPU power FIX: dirac-install: Make hashlib optional again (for previous versions of python, since the pilot may end up on old machines) *Framework FIX: allow to install agents with non-standard names (different from the module name) *DMS CHANGE: Consider files to reschedule and submit when they are Failed in FTS *WMS CHANGE: Move getCEStatus function back to using the ARC API [v6r14p11] *Core FIX: XXXTimeLeft - set limit to CPU lower than wall clock if unknown FIX: Logger - fix exception printing in gLogger.exception() CHANGE: InstallTools - added more info about the process in getStartupComponentStatus() CHANGE: Time - better report from timeThis() decorator *DMS CHANGE: FTSAgent - wait some time between 2 monitorings of each job *WMS NEW: pilotCommands - added CheckCECapabilities, CheckWNCapabilities commands NEW: Added dirac-wms-get-wn-parameters command *TS NEW: Added dirac-production-runjoblocal command FIX: TransformationAgent(Plugin) - clean getNextSite() and normalizeShares() FIX: TransformationPlugin - added setParameters() method *RSS FIX: dirac-rss-sync - move imports to after the Script.getPositionalArguments() *Resources NEW: Added dirac-resource-get-parameters command [v6r14p10] *Configuration FIX: Resources - getQueue() is fixed to get properly Tag parameters *Framework FIX: SecurityFileLog - fix for zipping very large files *Resources NEW: added dirac-resource-get-parameters command *WMS NEW: JobMonitoringHandler - add getJobsParameters() method NEW: pilotCommands - added CheckCECapabilities, CheckWNCapabilities NEW: Added dirac-wms-get-wn-parameters command NEW: Matcher - generate internal tags for MaxRAM and NumberOfProcessors parameters CHANGE: SiteDirector does not pass Tags to the Pilot FIX: Matcher(Handler) - do not send error log message if No match found, fixed Matcher return value not correctly interpreted [v6r14p9] *Core FIX: BaseClient - enhance retry connection logic to minimize the overall delay FIX: MessageBroker - fix of calling private __remove() method from outside of the class *Framework BUGFIX: dirac-(un)install-component - bug in importing InstallTools module *WMS: FIX: JobWrapper - fix in getting the OutputPath defined in the job *Resources FIX: ARCComputingElement - add queue to the XRSL string [v6r14p8] *Core FIX: XXXTimeLeft - minor fixes plus added the corresponding Test case FIX: ReturnValues - fixes in the doc strings to comply with the sphinx syntax FIX: SocketInfoFactory - in __sockConnect() catch exception when creating a socket *Interfaces FIX: Job.py - fixes in the doc strings to comply with the sphinx syntax *RSS NEW: Configurations.py - new possible configuration options for Downtime Policies *WMS CHANGE: StatesAccountingAgent - retry once and empty the local messages cache in case of failure to avoid large backlog of messages CHANGE: SiteDirector - do not send SharedArea and ClientPlatform as pilot invocation arguments CHANGE: Matcher - allow matching by hosts in multi-VO installations [v6r14p7] *Core CHANGE: XXXTimeLeft utilities revisited - all return real seconds, code refactoring - use consistently always the same CPU power *WMS FIX: JobAgent - code refactoring for the timeLeft logic part *Resources BUGFIX: ComputingElement - get rid of legacy getResourcesDict() call [v6r14p6] *Configuration FIX: Bdii2CSAgent - refresh configuration from Master before updating FIX: Bdii2CSAgent - distinguish the CE and the Cluster in the Glue 1.0 schema *DMS CHANGE: FTSAgent - make the amount of scheduled requests fetched by the FTSAgent a parameter in the CS CHANGE: RMS Operations - check whether the always banned policy is applied for SEs to a given access type *RMS FIX: RequestClient(DB,Manager) - fix bulk requests, lock the lines when selecting the requests to be assigned, update the LastUpdate time, and expose the assigned flag to the client *WMS FIX: JobAgent - when the application finishes with errors but the agent continues to take jobs, the timeLeft was not evaluated FIX: JobAgent - the initial timeLeft value was always set to 0.0 [v6r14p5] *Core FIX: X509Certificate - protect from VOMS attributes that are not decodable *Resources FIX: GFAL2_StorageBase - fixed indentation and a debug log typo *WMS BUGFIX: Matcher - only the first job was associated with the given pilot FIX: pilotTools - 0o22 is only a valid int for recent python interpreters, replaced by 18 [v6r14p4] *Core FIX: DictCache - fix the exception in the destructor preventing the final cache cleaning *Framework FIX: SystemAdministratorClientCLI - corrected info line inviting to update the pilot version after the software update *DMS FIX: FTSAgent - Add recovery of FTS files that can be left in weird statuses when the agent dies CHANGE: DataManager - allow to not get URLs of the replicas CHANGE: FTSJob - keep and reuse the FTS3 Context object *Storage CHANGE: StorageManagerClient - don't fail getting metadata for staging if at least one staged replica found *WMS FIX: CPUNormalization - protect MJF from 0 logical cores FIX: JobScheduling - fix printout that was saying "single site" and "multiple sites" in two consecutive lines NEW: pilotTools,Commands - added CEType argument, e.g. to specify Pool CE usage FIX: WatchDog - added checks of function return status, added hmsCPU initialization to 0, removed extra printout *Resources FIX: GFAL2 plugins - multiple bug fixes [v6r14p3] *Core BUGFIX: small bug fixed in dirac-install-component, dirac-uninstall-component BUGFIX: VOMS - remove the temporary file created when issuing getVOMSProxyInfo FIX: FileHelper - support unicode file names FIX: DictCache - purges all the entry of the DictCache when deleting the DictCache object *Framework BUGFIX: dirac-populate-component-db - avoid return statement out of scope *Interfaces BUGFIX: Dirac - in submitJob() faulty use of os.open *WMS FIX: JobWrapper - avoid evaluation of OutputData to [''] FIX: Matcher - the Matcher object uses a VO dependent Operations helper CHANGE: JobAgent - stop agent if time left is too small (default 1000 HS06.s) FIX: CPUNormalization - use correct denominator to get power in MJF *Resources FIX: ARCComputingElement - changed implementation of ldap query for getCEStatus [v6r14p2] *Core FIX: Use GSI version 0.6.3 by default CHANGE: Time - print out the caller information in the timed decorator CHANGE: dirac-install - set up ARC_PLUGIN_PATH environment variable *Framework FIX: dirac-proxy-info - use actimeleft VOMS attribute *Accounting CHANGE: Removed SRMSpaceTokenDeployment Accounting type *RSS CHANGE: ResourceStatus - re-try few times to update the RSS SE cache before giving up FIX: XXXCommand, XXXAction - use self.lof instead of gLogger CHANGE: Added support for all protocols for SEs managed by RSS *RMS FIX: Request - produce enhanced digest string FIX: RequestDB - fix in getDigest() in case of errors while getting request *Resources CHANGE: Propagate hideExceptions flag to the ObjectLoader when creating StorageElements FIX: ARCComputingElement - multiple fixes after experience in production *WMS FIX: Pilot commands - fixed an important bug, when using the dirac-wms-cpu-normalization script [v6r14p1] The version is buggy when used in pilots *Core NEW: dirac-install-component command replacing dirac-install-agent/service/executor commands *Resources NEW: FileStorage - plugin for "file" protocol FIX: ARCComputingElement - evaluate as int the job exit code *RSS FIX: CSHelpers - several fixes and beautifications [v6r14] *Core NEW: CSGlobals - includes Extensions class to consistently check the returned list of extensions with proper names NEW: ProxyManagerXXX, ProxyGeneration, X509XXX - support for RFC proxies NEW: ProxyInfo - VOMS proxy information without using voms commands NEW: LocalConfiguration - option to print out license information FIX: SocketInfo.py - check the CRL lists while handshaking Configuration NEW: ConfigurationClient - added getSectionTree() method *Framework NEW: InstalledComponentsDB will now store information about the user who did the installation/uninstallation of components. *Resources NEW: ARCComputingElement based on the ARC python API *RSS FIX: Improved logging all over the place *DMS NEW: New FileCatalog SecurityManager with access control based on policies, VOMSPolicy as one of the policy implementations. NEW: lfc_dfc_db_copy - script used by LHCb to migrate from the LFC to the DFC with Foreign Keys and Stored Procedures by accessing the databases directly NEW: FileManagerPs.py - added _getFileLFNs() to serve info for the Web Portal CHANGE: Moving several tests to TestDIRAC *Interfaces CHANGE: use jobDescription.xml as a StringIO object to avoid multiple disk write operations while massive job submission *WMS FIX: Watchdog - review for style and pylint CHANGE: Review of the Matcher code, extracting Limiter and Matcher as standalone utilities *Transformation NEW: New ported plugins from LHCb, added unit tests [v6r13p21] *TS FIX: Registering TargetSE for Standard TransformationAgent plugin [v6r13p20] *DMS FIX: DMSHelpers - allow for more than one Site defined to be local per SE *Resources FIX: XRootStorage - fix in getURLBase() [v6r13p19] FIX: changes incorporated from v6r12p53 patch [v6r13p18] *WMS FIX: JobWrapper - ported back from v6r14p9 the fix for getting OutputPath [v6r13p17] FIX: changes incorporated from v6r12p52 patch [v6r13p16] FIX: changes incorporated from v6r12p51 patch [v6r13p15] Included patches from v6r12p50 release [v6r13p14] *DMS FIX: ReplicateAndRegister - fix a problem when a file is set Problematic in the FC but indeed doesn't exist at all *Resources CHANGE: StorageFactory - enhance the logic of BaseSE inheritance in the SE definition in the CS *WMS CHANGE: CPUNormalization, dirac-wms-cpu-normalization - reading CPU power from MJF for comparison with the DIRAC evaluation FIX: SiteDirector - create pilot working directory in the batch system working directory and not in "/tmp" [v6r13p13] *DMS BUGFIX: FileCatalogClient - bug fixed in getDirectoryMetadata() [v6r13p12] *Resources FIX: StorageElement - bug fixed in inValid() CHANGE: StorageFactory - do not interpret VO parameter as mandatory [v6r13p11] *DMS BUGFIX: RemoveReplica - fix in singleRemoval() FIX: dirac-dms-user-lfns - increased timeout [v6r13p10] CHANGE: Use sublogger to better identify log source in multiple places *Core CHANGE: Review / beautify code in TimeLeft and LSFTimeLeft FIX: LSFTimeLeft - is setting shell variables, not environment variables, therefore added an "export" command to get the relevant variable and extract then the correct normalization *Accounting FIX: DataOperationPlotter - add better names to the data operations *DMS: FIX: DataManager - add mandatory vo parameter in __SEActive() CHANGE: dirac-dms-replicate-and-register-request - submit multiple requests to avoid too many files in a single FTS request FIX: FileCatalog - typo in getDirectoryMetadata() FIX: FileCatalog - pass directory name to getDirectoryMetadata and not file name FIX: DataManager - in __SEActive() break LFN list in smaller chunks when getting replicas from a catalog *WMS FIX: WMSAdministratorHandler - fix in reporting pilot statistics FIX: JobScheduling - fix in __getSitesRequired() when calling self.jobLog.info CHANGE: pilotCommands - when exiting with error, print out current processes info [v6r13p9] *Framework FIX: SystemLoggingDB - schema change for ClientIPs table to store IPv6 addresses *DMS BUGFIX: DMSRequestOperationsBase - bug fix in checkSEsRSS() FIX: RemoveFile - in __call__(): bug fix; fix in the BannedSE treatment logic *RMS BUGFIX: Operation - in catalogList() BUGFIX: ReqClient - in printOperation() *Resources FIX: GFAL2_StorageBase - added Lost, Cached, Unavailable in getSingleFileMetadata() output BUGFIX: GFAL2_StorageBase - fixed URL construction in put(get)SingleFile() methods *WMS FIX: InputDataByProtocol - removed StorageElement object caching [v6r13p8] *Framework FIX: MonitoringUtilities - minor bug fix *DMS FIX: DataManager - remove local file when doing two hops transfer *WMS FIX: SandboxStoreClient - get the VO info from the delegatedGroup argument to use for the StorageElement instantiation *TMS CHANGE: Transformation(Client,DB,Manager) - multiple code clean-up without changing the logic [v6r13p7] *Core NEW: X509CRL - class to handle certificate revocation lists *DMS FIX: RequestOperations/RemoveFile.py - check target SEs to be online before performing the removal operation. FIX: SecurityManager, VOMSPolicy - make the vomspolicy compatible with the old client by calling in case of need the old SecurityManager *Resources BUGFIX: Torque, GE - methods must return Message field in case of non-zero return status FIX: SRM2Storage - when used internaly, listDirectory should return urls and not lfns *WMS FIX: ConfigureCPURequirements pilot command - add queue CPU length to the extra local configuration FIX: JobWrapper - load extra local configuration of any *RMS FIX: RequestDB - fix in getRequestSummaryWeb() to suit the Web Portal requirements *Transformation FIX: TransformationManagerHandler - fix in getTransformationSummaryWeb() to suit the Web Portal requirements [v6r13p6] *Core FIX: X509Chain - use SHA1 signature encryption in all tha cases *Resources FIX: ComputingElement - take CPUTime from its configuration defined in the pilot parameters *WMS FIX: SiteDirector - correctly configure jobExecDir and httpProxy Queue parameters [v6r13p5] *Resources BUGFIX: Torque - getCEStatus() must return integer job numbers FIX: StorageBase - removed checking the VO name inside the LFN *WMS FIX: InputData, JobScheduling - StorageElement needs to know its VO *DMS FIX: ReplicateAndRegister - Add checksumType to RMS files when adding checksum value FIX: DataManager - remove unnecessary access to RSS and use SE.getStatus() FIX: DMHelpers - take into account Alias and BaseSE in site-SE relation *RMS FIX: Request - bug fixed in optimize() in File reassignment from one Operation to another *Transformation FIX: TransformationDB - set derived transformation to Automatic [v6r13p4] *Core FIX: VOMSService - treat properly the case when the VOMS service returns no result in attGetUserNickname() *DMS FIX: FTSAgent, ReplicateAndRegister - make sure we use source replicas with correct checksum *RMS FIX: Request - minor fix in setting the Request properties, suppressing pylint warnings CHANGE: File, Reques, Operation, RequestDB - remove the use of sqlalchemy on the client side *Resources FIX: StorageElement - import FileCatalog class rather than the corresponding module FIX: SLURM - proper formatting commands using %j, %T placeholders FIX: SSHComputingElement - return full job references from getJobStatus() *RSS FIX: DowntimeCommand - checking for downtimes including the time to start in hours *Workflow CHANGE: FailoverRequest - assign to properties rather than using setters *Transformation FIX: TransformationClient(DB,Utilities) - fixes to make derived transformations work [v6r13p3] *DMS FIX: DataManager - in putAndRegister() specify explicitly registration protocol to ensure the file URL available right after the transfer *Resources FIX: SRM2Storage - use the proper se.getStatus() interface ( not the one of the RSS ) [v6r13p2] *Framework FIX: SystemAdministratorHandler - install WebAppDIRAC extension only in case of Web Portal installation CHANGE: dirac-populate-component-db - check the setup of the hosts to register into the DB only installations from the same setup; check the MySQL installation before retrieving the database information *DMS FIX: FTSAgent - fix in parsing the server result FIX: FTSFile - added Waiting status FIX: FTSJob - updated regexps for the "missing source" reports from the server; more logging message *Resources FIX: SRM2Storage - fix in treating the checksum type FIX: StorageElement - removed getTransportURL from read methods *RMS FIX: Request - typo in the optimize() method [v6r13p1] *Framework CHANGE: SystemAdminstratorIntegrator - can take a list of hosts to exclude from contacting *DMS FIX: DataManager - fix in __getFile() in resolving local SEs FIX: dirac-dms-user-lfns - sort result, simplify logic *RMS FIX: Request - Use DMSHelper to resolve the Failovers SEs FIX: Operation - treat the case where the SourceSE is None *WMS FIX: WMSAdministratorHandler - return per DN dictionary from getPilotStatistics [v6r13] CHANGE: Separating fixed and variable parts of error log messages for multiple systems to allow SystemLogging to work *Core FIX: MySQL.py - treat in detailed way datetime functions in __escapeString() FIX: DictCache.get() returns now None instead of False if no or expired value NEW: InstallTools - allow to define environment variables to be added to the component runit run script NEW: Changes to make the DISET protocol IP V6 ready CHANGE: BaseClient - retry service call on another instance in case of failure CHANGE: InnerRPCClient - retry 3 times in case of exception in the transport layer CHANGE: SocketInfo - retry 3 times in case of handshaking error CHANGE: MySQL - possibility to specify charset in the table definition FIX: dirac-install, dirac-distribution - removed obsoleted defaults NEW: Proxy utility module with executeWithUserProxy decorator function *Configuration NEW: CSAPI,dirac-admin-add-shifter - function, and script, for adding or modifying a shifter in the CS *Framework FIX: NotificationDB - escape fields for sorting in getNotifications() NEW: Database, Service, Client, commands for tracking the installed DIRAC components *Interfaces CHANGE: Dirac - changed method names, keeping backward compatibility CHANGE: multiple commands updated to use the new Dirac API method names *DMS NEW: Native use of the FTS3 services CHANGE: Removed the use of current DataLogging service CHANGE: DataManager - changes to manage URLs inside StorageElement objects only FIX: DataManager - define SEGroup as accessible at a site CHANGE: DirectoryListing - extracted from FileCatalogClientCLI as an independent utility CHANGE: MetaQuery - extracted from FileCatalogClientCLI as an independent utility CHANGE: FileCatalogClientCLI uses external DirectoryListing, MetaQuery utilities CHANGE: FileCatalog - replace getDirectoryMetadata by getDirectoryUserMetadata NEW: FileCatalog - added new getDirectoryMetadata() interface to get standard directory metadata NEW: FileCatalog - possibility to find files by standard metadata NEW: FileCatalog - possibility to use wildcards in the metadata values for queries NEW: DMSHelpers class NEW: dirac-dms-find-lfns command *WMS NEW: SiteDirector - support for the MaxRAM queue description parameter CHANGE: JobScheduling executor uses the job owner proxy to evaluate which files to stage FIX: DownloadInputData - localFile was not defined properly FIX: DownloadInputData - could not find cached files (missing [lfn]) *RMS CHANGE: Removed files from the previous generation RMS CHANGE: RMS refactored based on SQLAlchemy NEW: ReqClient - added options to putRequest(): useFailoverProxy and retryMainServer CHANGE: DMSRequestOperationsBase - delay execution or cancel request based on SE statuses from RSS/CS FIX: Fixes to make use of RequestID as a unique identifier. RequestName can be used in commands in case of its uniqueness *Resources NEW: Computing - BatchSystem classes introduced to be used both in Local and SSH Computing Elements CHANGE: Storage - reworked Storage Element/Plugins to encapsulate physical URLs NEW: GFAL2_StorageBase.py, GFAL2_SRM2Storage.py, GFAL2_XROOTStorage.py *RSS: NEW: dirac-admin-allow(ban)-se - added RemoveAccess status CHANGE: TokenAgent - added more info to the mail *TS CHANGE: Task Manager plugins [v6r12p53] *DMS CHANGE: FileCatalogClientCLI - ls order by size, human readable size value FIX: DirectoryMetadata - enhanced error message in getDirectoryMetadata *WMS BUGFIX: JobAgent - bug when rescheduling job due to glexec failure *TS NEW: TransformationCLI - added getOutputFiles, getAllByUser commands NEW: Transformation - added getAuthorDNfromProxy, getTransformationsByUser methods *Resources CHANGE: GlobusComputingElement - simplify creating of pilotStamp [v6r12p52] *DMS NEW: dirac-dms-directory-sync - new command to synchronize the contents of a local and remote directories FIX: DataManager - in removeFile() return successfully if empty input file list *TS NEW: TransformationCLI - getInputDataQuery command returning inputDataQuery of a given transformation [v6r12p51] *Core FIX: dirac-install - fix to work with python version prior to 2.5 *DMS CHANGE: FileCatalogClientCLI - possibility to set multiple metadata with one command *Resources FIX: HTCondorComputingElement - multiple improvements [v6r12p50] *Core FIX: dirac-install - define TERMINFO variable to include local sources as well *Framework FIX: SystemAdministratorHandler - show also executors in the log overview *DMS FIX: FileCatalogClientCLI - use getPath utility systematically to normalize the paths passed by users *WMS FIX: PilotStatusAgent - split dynamic and static parts in the log error message *Resources NEW: HTCondorCEComputingElement class [v6r12p49] *Resources FIX: GlobusComputingElement - in killJob added -f switch to globus-job-clean command FIX: ARCComputingElement - create working directory if it does not exist *DMS CHANGE: DataManager - added XROOTD to registration protocols *TMS FIX: TransformationCLI - doc string [v6r12p48] *DMS FIX: DirectoryTreeBase - fix in changeDirectoryXXX methods to properly interpret input [v6r12p47] *DMS BUGFIX: FileCatalogClientCLI - wrong signature in the removeMetadata() service call [v6r12p46] *Core FIX: GraphData - check for missing keys in parsed_data in initialize() *WMS CHANGE: PilotStatusAgent - kill pilots being deleted; do not delete pilots still running jobs *RSS CHANGE: Instantiate RequestManagementDB/Client taking into account possible extensions *Resources FIX: GlobusComputingElement - evaluate WaitingJobs in getCEStatus() FIX: SRM2Storage - error 16 of exists call is interpreted as existing file FIX: XROOTStorage - added Lost, Cached, Unavailable in the output of getSingleMetadata() *WMS FIX: pilotCommands - removed unnecessary doOSG() function [v6r12p45] *Resources FIX: SRM2Storage - error 22 of exists call is interpreted as existing file ( backport from v6r13 ) [v6r12p44] *WMS FIX: SiteDirector - consider also pilots in Waiting status when evaluating queue slots available *Resources NEW: SRM2Storage - makes use of /Resources/StorageElements/SRMBusyFilesExist option to set up the mode of interpreting the 22 error code as existing file [v6r12p43] *DMS: FIX: DirectoryTreeBase - avoid double definition of FC_DirectoryUsage table in _rebuildDirectoryUsage() [v6r12p42] FIX: added fixes from v6r11p34 patch release [v6r12p41] *WMS CHANGE: dirac-wms-job-submit - "-r" switch to enable job repo [v6r12p40] *DMS FIX: DirectoryTreeBase.py - set database engine to InnoDB [v6r12p39] FIX: imported fixes from rel-v6r11 [v6r12p38] *DMS CHANGE: DataManager - enhanced real SE name resolution *RMS FIX: Request - fixed bug in the optimization of requests with failover operations *Resources CHANGE: StorageFactory - allow for BaseSE option in the SE definition [v6r12p37] *Core FIX: InstallTools - force $HOME/.my.cnf to be the only defaults file [v6r12p36] *Configuration FIX: Utilities.py - bug fix getSiteUpdates() [v6r12p35] *Core CHANGE: VOMSService - add URL for the method to get certificates *DMS FIX: DataManager - in __replicate() set do not pass file size to the SE if no third party transfer FIX: RemoveFile, ReplicateAndRegister - regular expression for "no replicas" common for both DFC and LFC *WMS FIX: WMSHistoryCorrector - make explicit error if no data returned from WMSHistory accounting query [v6r12p34] *DMS BUGFIX: FileCatalogWithFkAndPsDB - fix storage usage calculation [v6r12p33] *Core NEW: VOMSService - added method admListCertificates() *DMS BUGFIX: dirac-dms-put-and-register-request - missing Operation in the request *Resources FIX: sshce - better interpretation of the "ps" command output [v6r12p32] *RMS FIX: ReqManager - in getRequest() possibility to accept None type argument for any request [v6r12p31] *WMS FIX: pilotCommands - import json module only in case it is needed [v6r12p30] *Core FIX: InstallTools - 't' file is deployed for agents installation only FIX: GOCDBClient - creates unique DowntimeID using the ENDPOINT *Framework FIX: SystemAdministratorHandler - use WebAppDIRAC extension, not just WebApp *DMS: FIX: FileCatalogComponents.Utilities - do not allow empty LFN names in checkArgumentDict() [v6r12p29] *CS CHANGE: CSCLI - use readline to store and resurrect command history *WMS FIX: JobWrapper - bug fixed in the failoverTransfer() call CHANGE: dirac-wms-job-submit - added -f flag to store ids *DMS FIX: DataManager - make successful removeReplica if missing replica in one catalog *RMS FIX: Operation, Request - limit the length of the error message [v6r12p28] *RMS FIX: Request - do not optimize requests already in the DB [v6r12p27] *Core CHANGE: InstallTools - install "t" script to gracefully stop agents *DMS FIX: FileCatalog - return GUID in DirectoryParameters *Resource CHANGE: DFC/LFC clients - added setReplicaProblematic() [v6r12p26] *DMS BUGFIX: FileCatalog - getDirectoryMetadata was wrongly in ro_meta_methods list *RMS FIX: Operation - temporary fix in catalog names evaluation to smooth LFC->DFC migration - not to forget to remove afterwards ! *WMS CHANGE: JobWrapper - added MasterCatalogOnlyFlag configuration option [v6r12p25] *DMS BUGFIX: PutAndRegister, RegitserFile, RegisterReplica, ReplicateAndRegister - do not evaluate the catalog list if None [v6r12p24] *DMS: FIX: DataManager - retry RSS call 5 times - to be reviewed [v6r12p23] *DMS FIX: pass a catalog list to the DataManager methods FIX: FileCatalog - bug fixed in the catalog list evaluation [v6r12p22] *DMS FIX: RegisterFile, PutAndRegister - pass a list of catalogs to the DataManager instead of a comma separated string FIX: FTSJob - log when a job is not found in FTS CHANGE: dropped commands dirac-admin-allow(ban)-catalog *Interfaces CHANGE: Dirac, JobMonitoringHandler,dirac-wms-job-get-jdl - possibility to retrieve original JDL *WMS CHANGE: JobManifest - make MaxInputData a configurable option [v6r12p21] *RMS BUGFIX: File,Operation,RequestDB - bug making that the request would always show the current time for LastUpdate *WMS FIX: JobAgent - storing on disk retrieved job JDL as required by VMDIRAC ( to be reviewed ) [v6r12p20] *DMS FIX: DataManager - more informative log messages, checking return structure FIX: FileCatalog - make exists() behave like LFC file catalog client by checking the unicity of supplied GUID if any FIX: StorageElementProxyHandler - do not remove the cache directory *Framework FIX: SystemAdministratorClient - increase the timeout to 300 for the software update *RMS FIX: Operation.py - set Operation Scheduled if one file is Scheduled CHANGE: Request - group ReplicateAndRegister operations together for failover requests: it allows to launch all FTS jobs at once *Resources FIX: LcgFileCatalogClient - fix longstanding problem in LFC when several files were not available (only one was returned) *TS BUGFIX: TransformationCleaning,ValidateOutputDataAgent - interpret correctly the result of getTransformationParameters() call FIX: TaskManager - fix exception in RequestTaskAgent [v6r12p19] *Core FIX: Core.py - check return value of getRecursive() call *DMS FIX: FileCatalog - directory removal is successful if does not exist special treatment of Delete operation *WMS FIX: InputDataByProtocol - fix interpretation of return values [v6r12p18] *DMS FIX: FTSStrategy - config option name FIX: DataManager - removing dirac_directory flag file only of it is there in __cleanDirectory() *RMS FIX: Operation - MAX_FILES limit set to 10000 FIX: ReqClient - enhanced log messages *TMS FIX: TaskManager - enhanced log messages *RSS FIX: DowntimeCommand - fixed mix of SRM.NEARLINE and SRM *WMS FIX: InputDataByProtocol - fixed return structure [v6r12p16] *DMS FIX: IRODSStorageElement more complete implementation FIX: FileCatalogHandler(DB) - make removeMetadata bulk method *Resources FIX: FileCatalog - make a special option CatalogList (Operations) to specify catalogs used by a given VO [v6r12p15] *Core FIX: ProcessPool - kill the working process in case of the task timeout FIX: FileHelper - count transfered bytes in DataSourceToNetwork() *DMS BUGFIX: FileCatalogCLI - changed interface in changePathXXX() methods NEW: IRODSStorageElementHandler class CHANGE: FileCatalog - separate metadata and file catalog methods, apply metadata methods only to Metadata Catalogs *Resources FIX: SSHTorqueComputingElement - check the status of the ssh call for qstat *WMS FIX: WatchdogLinux - fixed typo [v6r12p14] *TS FIX: TaskManagerAgentBase: avoid race conditions when submitting to WMS *DMS NEW: FileCatalog - added new components ( directory tree, file manager ) making use of foreign keys and stored procedures FIX: DataManager returns properly the FileCatalog errors [v6r12p13] *TS BUGFIX: TransformationAgent - data member not defined *WMS FIX: InputData(Resolution,ByProtocol) - possibility to define RemoteProtocol [v6r12p12] *WMS BUGFIX: pilotTools - missing comma [v6r12p11] *WMS FIX: CPUNormalization - dealing with the case when the maxCPUTime is not set in the queue definition FIX: pilotTools - added option pilotCFGFile [v6r12p10] *DMS FIX: StorageElementProxy - BASE_PATH should be a full path *Resources FIX: SRM2Storage - return specific error in putFile *TS FIX: TransformationAgent - fix to avoid an exception in finalize and double printing when terminating the agent BUGFIX: TransformationDB - fix return value in setTransformationParameter() [v6r12p9] *Core CHANGE: SiteCEMapping - getSiteForCE can take site argu ment to avoid confusion *Interfaces FIX: Job - provide optional site name in setDestinationCE() *WMS FIX: pilotCommands - check properly the presence of extra cfg files when starting job agent FIX: JobAgent - can pick up local cfg file if extraOptions are specified [v6r12p8] *Core FIX: dirac-configure - correctly deleting useServerCertificate flag BUGFIX: InstallTools - in fixMySQLScript() *DMS BUGFIX: DatasetManager - bug fixes CHANGE: StorageElementProxy - internal SE object created with the VO of the requester *TS FIX: dirac-transformation-xxx commands - do not check the transformation status CHANGE: Agents - do not use shifter proxy FIX: TransformationAgent - correct handling of replica cache for transformations when there were more files in the transformation than accepted to be executed FIX: TransformationAgent - do not get replicas for the Removal transformations *RMS NEW: new SetFileStatus Operation [v6r12p7] *Core FIX: dirac-configure - always removing the UseServerCertificate flag before leaving FIX: ProcessPool - one more check for the executing task ending properly *Interfaces FIX: Dirac.py - use printTable in loggingInfo() [v6r12p6] FIX: fixes from v6r11p26 patch release [v6r12p5] *Core FIX: VOMS.py - do not use obsoleted -dont-verify-ac flag with voms-proxy-info *TS FIX: TransformationManager - no status checked at level service [v6r12p4] FIX: fixes from v6r11p23 patch release [v6r12p3] *Configuration CHANGE: dirac-admin-add-resources - define VOPath/ option when adding new SE *Resources NEW: StorageFactory - modify protocol Path for VO specific value *DMS FIX: FileCatalog - check for empty input in checkArgumentFormat utility FIX: DataManager - protect against FC queries with empty input [v6r12p2] *Core FIX: dirac-install - svn.cern.ch rather than svnweb.cern.ch is now needed for direct HTTP access to files in SVN *WMS FIX: dirac-wms-cpu-normalization - when re-configuring, do not try to dump in the diracConfigFilePath [v6r12p1] *Configuration FIX: Core.Utilities.Grid, dirac-admin-add-resources - fix to make a best effort to guess the proper VO specific path of a new SE *WMS FIX: dirac-configure, pilotCommands, pilotTools - fixes to use server certificate [v6r12] *Core CHANGE: ProcessPool - do not stop working processes by default NEW: ReturnValue - added returnSingleResult() utility FIX: MySQL - correctly parse BooleanType FIX: dirac-install - use python 2.7 by default FIX: dirac-install-xxx commands - complement installation with the component setup in runit NEW: dirac-configure - added --SkipVOMSDownload switch, added --Output switch to define output configuration file CHANGE: ProcessPool - exit from the working process if a task execution timed out NEW: ProcessMonitor - added evaluation of the memory consumed by a process and its children NEW: InstallTools - added flag to require MySQL installation FIX: InstallTools - correctly installing DBs extended (with sql to be sourced) FIX: InstallTools - run MySQL commands one by one when creating a new database FIX: InstallTools - fixMySQLScripts() fixes the mysql start script to ognore /etc/my.cnf file CHANGE: Os.py - the use of "which" is replaced by distutils.spawn.find_executable NEW: Grid.py - ldapSA replaced by ldapSE, added getBdiiSE(CE)Info() methods CHANGE: CFG.py - only lines starting with ^\s*# will be treated as comments CHANGE: Shifter - Agents will now have longer proxies cached to prevent errors for heavy duty agents, closes #2110 NEW: Bdii2CSAgent - reworked to apply also for SEs and use the same utilities for the corresponding command line tool NEW: dirac-admin-add-resources - an interactive tool to add and update sites, CEs, SEs to the DIRAC CS CHANGE: dirac-proxy-init - added message in case of impossibility to add VOMS extension FIX: GOCDBClient - handle correctly the case of multiple elements in the same DT *Accounting NEW: Allow to have more than one DB for accounting CHANGE: Accounting - use TypeLoader to load plotters *Framework FIX: Logger - fix FileBackend implementation *WMS NEW: Refactored pilots ( dirac-pilot-2 ) to become modular following RFC #18, added pilotCommands.py, SiteDirector modified accordingly CHANGE: InputData(Executor) - use VO specific catalogs NEW: JobWrapper, Watchdog - monitor memory consumption by the job ( in a Warning mode ) FIX: SandboxStoreHandler - treat the case of exception while cleaning sandboxes CHANGE: JobCleaningAgent - the delays of job removals become CS parameters BUGFIX: JobDB - %j placeholder not replaced after rescheduling FIX: JobDB - in the SQL schema description reorder tables to allow foreign keys BUGFIX: JobAgent, Matcher - logical bug in using PilotInfoReported flag FIX: OptimizerExecutor - when a job fails the optimization chain set the minor status to the optimiser name and the app status to the fail error *Resources NEW: StorageElement - added a cache of already created SE objects CHANGE: SSHTorqueComputingElement - mv getCEStatus to remote script *ResourceStatus NEW: ResourceManagementClient/DB, DowntimeCommand - distinguish Disk and Tape storage FIX: GODDBClient - downTimeXMLParsing() can now handle the "service type" parameter properly CHANGE: dirac-rss-xxx commands use the printTable standard utility FIX: dirac-dms-ftsdb-summary - bug fix for #2096 *DMS NEW: DataManager - add masterCatalogOnly flag in the constructor FIX: DataManager - fix to protect against non valid SE CHANGE: FC.DirectoryLevelTree - use SELECT ... FOR UPDATE lock in makeDir() FIX: FileCatalog - fixes in using file and replica status CHANGE: DataManager - added a new argument to the constructor - vo CHANGE: DataManager - removed removeCatalogFile() and dirac-dms-remove-catalog-file adjusted CHANGE: Several components - field/parameter CheckSumType all changed to ChecksumType CHANGE: PoolXMLCatalog - add the SE by default in the xml dump and use the XML library for dumping the XML FIX: XROOTStorageElement - fixes to comply with the interface formalism *SMS FIX: StorageManagementDB - small bugfix to avoid SQL errors *RMS NEW: Added 'since' and 'until' parameters for getting requests NEW: Request - added optimize() method to merge similar operations when first inserting the request NEW: ReqClient, RequestDB - added getBulkRequest() interface. RequestExecutingAgent can use it controlled by a special flag FIX: Operation, Request - set LastUpdate time stamp when reaching final state FIX: OperationHandlerBase - don't erase the original message when reaching the max attempts FIX: removed some deprecated codes FIX: RequestTask - always set useServerCerificate flag to tru in case of executing inside an agent CHANGE: gRequestValidator removed to avoid object instantiation at import NEW: dirac-rms-cancel-request command and related additions to the db and service classes *TMS NEW: WorkflowTaskAgent is now multi-threaded NEW: Better use of threads in Transformation Agents CHANGE: TransformationDB - modified such that the body in a transformation can be updated FIX: TransformationCleaningAgent - removed non-ASCII characters in a comment [v6r11p34] *Resources NEW: GlobusComputingElement class [v6r11p33] *Configuration FIX: Resources - avoid white spaces in OSCompatibility [v6r11p32] *Core CHANGE: BaseClient, SSLSocketFactory, SocketInfo - enable TLSv1 for outgoing connections via suds, possibility to configure SSL connection details per host/IP [v6r11p31] *Core FIX: CFG - bug fixed in loadFromBuffer() resulting in a loss of comments *Resources FIX: SSHTorqueComputingElement - check the status of ssh call for qstat *DMS FIX: FileCatalog - return LFN name instead of True from exists() call if LFN already in the catalog [v6r11p30] *DMS CHANGE: FileCatalogCLI - add new -D flag for find to print only directories [v6r11p29] *DMS FIX: FTS(Agent,Startegy,Gragh) - make use of MaxActiveJobs parameter, bug fixes *TMS FIX: Transformation(Agent,Client) - Operations CS parameters can be defined for each plugin: MaxFiles, SortedBy, NoUnusedDelay. Fixes to facilitate work with large numbers of files. [v6r11p28] *Core FIX: InstallTools - check properly the module availability before installation *WMS FIX: JobScheduling - protection against missing dict field RescheduleCounter *TMS FIX: TransformationCleaningAgent - execute DM operations with the shifter proxy [v6r11p27] *Core BUGFIX: InstallTools - bug fix in installNewPortal() *WMS FIX: Watchdog - disallow cputime and wallclock to be negative *TS FIX: TransformationAgent - correct handling of replica caches when more than 5000 files BUGFIX: ModuleBase - bug fix in execute() BUGFIX: Workflow - bug fix in createStepInstance() *DMS BUGFIX: DiractoryTreeBase - bug fix in getDirectoryPhysicalSizeFromUsage() *Resources FIX: XROOTStorage - back ported fixes from #2126: putFile would place file in the wrong location on eos [v6r11p26] *Framework FIX: UserProfileDB.py - add PublishAccess field to the UserProfileDB *RSS FIX: Synchronizer.py - fix deletion of old resources *DMS FIX: DataManager - allow that permissions are OK for part of a list of LFNs ( __verifyWritePermission() ) (when testing write access to parent directory). Allows removal of replicas even if one cannot be removed FIX: DataManager - test SE validity before removing replica *RMS FIX: RequestTask - fail requests for users who are no longer in the system FIX: RequestExecutingAgent - fix request timeout computation [v6r11p25] *Interfaces FIX: Job.py - bring back different logfile names if they have not been specified by the user [v6r11p24] *DMS BUGFIX: SEManagerDB - bug fixed in getting connection in __add/__removeSE [v6r11p23] *DMS CHANGE: FTSRequest is left only to support dirac-dms-fts-XXX commands [v6r11p22] *DMS FIX: FTSJob - fixes in the glite-transfer-status command outpu parsing FIX: TransformationClient - allow single lfn in setFileStatusForTransformation() *WMS FIX: StatesMonitoringAgent - install pika on the fly as a temporary solution [v6r11p21] *DMS BUGFIX: dirac-dms-remove-replicas - continue in case of single replica failure FIX: dirac-rms-xxx scripts - use Script.getPositionalArgs() instead of sys.argv *Workflow FIX: Test_Modules.py - fix in mocking functions, less verbose logging [v6r11p20] *DMS BUGFIX: DataManager - in __SEActive() use resolved SE name to deal with aliases BUGFIX: FileMetadata - multiple bugs in __buildUserMetaQuery() [v6r11p19] *DMS FIX: FTSJob - fix FTS job monitoring a la FTS2 *RMS CHANGE: ReqClient - added setServer() method FIX: File,Operation,Request - call the getters to fetch the up-to-date information from the parent [v6r11p18] *DMS FIX: FTSAgent(Job) - fixes for transfers requiring staging (bringOnline) and adaptation to the FTS3 interface *WMS FIX: StatesMonitoringAgent - resend the records in case of failure [v6r11p17] *DMS FIX: FileCatalog - in multi-VO case get common catalogs if even VO is not specified *Resources FIX: ComputintgElement - bugfix in available() method *WMS FIX: SiteDirector - if not pilots registered in the DB, pass empty list to the ce.available() [v6r11p16] *RMS BUGFIX: Request,Operation,File - do not cast to str None values [v6r11p15] *DMS FIX: ReplicateAndRegister - do not create FTSClient if no FTSMode requested CHANGE: FTSAgent(Job,File) - allow to define the FTS2 submission command; added --copy-pin-lifetime only for a tape backend parse output of both commands (FTS2, FTS3) consider additional state for FTS retry (Canceled) *RMS FIX: Operation, Request - treat updates specially for Error fields *TMS FIX: TransformationAgent - fixes in preparing json serialization of requests *WMS NEW: StateMonitoringAgent - sends WMS history data through MQ messages [v6r11p14] *WMS CHANGE: JobDB - removed unused tables and methods CHANGE: removed obsoleted tests *DMS FIX: FTSAgent - recover case when a target is not in FTSDB CHANGE: FTSAgent(Job) - give possibility to specify a pin life time in CS *RMS FIX: Make RMS objects comply with Python Data Model by adding __nonzero__ methods [v6r11p13] *DMS BUGFIX: SEManager - in SEManagerDB.__addSE() bad _getConnection call, closes #2062 [v6r11p12] *Resources CHANGE: ARCComputingElement - accomodate changes in the ARC job reported states *Configuration CHANGE: Resources - define a default FTS server in the CS (only for v6r11 and v6r12) *DMS FIX: FTSStrategy - allow to use a given channel more than once in a tree FIX: FTSAgent - remove request from cache if not found FIX: FTSAgent - recover deadlock situations when FTS Files had not been correctly updated or were not in the DB *RMS FIX: RequestExecutingAgent - fix a race condition (cache was cleared after the request was put) FIX: RequestValidator - check that the Operation handlers are defined when inserting a request [v6r11p11] *Core FIX: TransportPool - fixed exception due to uninitialized variable FIX: HTTPDISETSocket - readline() takes optional argument size ( = 0 ) *DMS FIX: FTSAgent - check the type of the Operation object ( can be None ) and some other protections FIX: FTSClient - avoid duplicates in the file list *RMS FIX: ReqClient - modified log message CHANGE: dirac-dms-fts-monitor - allow multiple comma separated LFNs in the arguments [v6r11p10] *RSS FIX: DowntimeCommand, Test_RSS_Command_GOCDBStatusCommand - correctly interpreting list of downtimes *RMS FIX: ReplicateAndRegister - Create a RegisterReplica (not RegisterFile) if ReplicateAndRegister fails to register FIX: OperationHandlerBase - handle correctly Attempt counters when SEs are banned FIX: ReplicateAndRegister - use FC checksum in case of mismatch request/PFN FIX: FTSAgent - in case a file is Submitted but the FTSJob is unknown, resubmit FIX: FTSAgent - log exceptions and put request to DB in case of exception FIX: FTSAgent - handle FTS error "Unknown transfer state NOT_USED", due to same file registered twice (to be fixed in RMS, not clear origin) *WMS FIX: JobStateUpdateHandler - status not updated while jobLogging is, due to time skew between WN and DB service FIX: JobStateUpdateHandler - stager callback not getting the correct status Staging (retry for 10 seconds) [v6r11p9] *Core NEW: AgentModule - set AGENT_WORKDIRECTORY env variable with the workDirectory NEW: InstallTools - added methods for the new web portal installation *DMS FIX: ReplicateAndRegister - apply same error logic for DM replication as for FTS *Resources: FIX: SRM2Storage - fix log message level FIX: SRM2Storage - avoid useless existence checks *RMS FIX: ForwardDISET - a temporary fix for a special LHCb case, to be removed asap FIX: ReqClient - prettyPrint is even prettier FIX: RequestTask - always use server certificates when executed within an agent [v6r11p8] *TMS FIX: TransformationDB - fix default value within ON DUPLICATE KEY UPDATE mysql statement [v6r11p7] *Framework BUGFIX: ProxyDB.py - bug in a MySQL table definition *DMS FIX: ReplicateAndRegister.py - FTS client is not instantiated in the c'tor as it might not be used, *WMS FIX: JobWrapper - don't delete the sandbox tar file if upload fails FIX: JobWrapper - fix in setting the failover request *RMS FIX: RequestDB - add protections when trying to get a non existing request [v6r11p6] *WMS FIX: InpudDataResolution - fix the case when some files only have a local replica FIX: DownloadInputData, InputDataByProtocol - fix the return structure of the execute() method *Resources NEW: LocalComputingElement, CondorComputingElement [v6r11p5] FIX: Incorporated changes from v6r10p25 patch *Framework NEW: Added getUserProfileNames() interface *WMS NEW: WMSAdministrator - added getPilotStatistics() interface BUGFIX: JobWrapperTemplate - use sendJobAccounting() instead of sendWMSAccounting() FIX: JobCleaningAgent - skip if no jobs to remove *DMS BUGFIX: FileCatalogClientCLI - bug fix in the metaquery construction *Resources CHANGE: StorageElement - enable Storage Element proxy configuration by protocol name *TMS NEW: TransformationManager - add Scheduled to task state for monitoring [v6r11p4] *Framework NEW: ProxyDB - added primary key to ProxyDB_Log table CHANGE: ProxyManagerHandler - purge logs once in 6 hours *DMS FIX: DataManager - fix in the accounting report for deletion operation CHANGE: FTSRequest - print FTS GUID when submitting request FIX: dirac-dms-fts-monitor - fix for using the new FTS structure FIX: DataLoggingDB - fix type of the StatusTimeOrder field FIX: DataLoggingDB - take into account empty date argument in addFileRecord() FIX: ReplicateAndRegister - use active replicas FIX: FTS related modules - multiple fixes *WMS NEW: SiteDirector - pass the list of already registered pilots to the CE.available() query FIX: JobCleaningAgent - do not attempt job removal if no eligible jobs *Resources FIX: LcgFileCatalogClient - if replica already exists while registration, reregister NEW: CREAM, SSH, ComputingElement - consider only registered pilots to evaluate queue occupancy [v6r11p3] FIX: import gMonitor from it is original location *Core FIX: FC.Utilities - treat properly the LFN names starting with /grid ( /gridpp case ) *Configuration FIX: LocalConfiguration - added exitCode optional argument to showHelp(), closes #1821 *WMS FIX: StalledJobAgent - extra checks when failing Completed jobs, closes #1944 FIX: JobState - added protection against absent job in getStatus(), closes #1853 [v6r11p2] *Core FIX: dirac-install - skip expectedBytes check if Content-Length not returned by server FIX: AgentModule - demote message "Cycle had an error:" to warning *Accounting FIX: BaseReporter - protect against division by zero *DMS CHANGE: FileCatalogClientCLI - quite "-q" option in find command FIX: DataManager - bug fix in __initializeReplication() FIX: DataManager - less verbose log message FIX: DataManager - report the size of removed files only for successfully removed ones FIX: File, FTSFile, FTSJob - SQL tables schema change: Size filed INTEGER -> BIGINT *RMS FIX: dirac-rms-reset-request, dirac-rms-show-request - fixes FIX: ForwardDISET - execute with trusted host certificate *Resources FIX: SSHComputingElement - SSHOptions are parsed at the wrong place NEW: ComputingElement - evaluate the number of available cores if relevant *WMS NEW: JobMonitoringHander - added export_getOwnerGroup() interface *TMS CHANGE: TransformationCleaningAgent - instantiation of clients moved in the initialize() [v6r11p1] *RMS FIX: ReqClient - failures due to banned sites are considered to be recoverable *DMS BUGFIX: dirac-dms-replicate-and-register-request - minor bug fixes *Resources FIX: InProcessComputingElement - stop proxy renewal thread for a finished payload [v6r11] *Core FIX: Client - fix in __getattr__() to provide dir() functionality CHANGE: dirac-configure - use Registry helper to get VOMS servers information BUGFIX: ObjectLoader - extensions must be looked up first for plug-ins CHANGE: Misc.py - removed obsoleted NEW: added returnSingleResult() generic utility by moving it from Resources/Utils module *Configuration CHANGE: Resources.getDIRACPlatform() returns a list of compatible DIRAC platforms NEW: Resources.getDIRACPlatforms() used to access platforms from /Resources/Computing/OSCompatibility section NEW: Registry - added getVOs() and getVOMSServerInfo() NEW: CE2CSAgent - added VO management *Accounting FIX: AccountingDB, Job - extra checks for invalid values *WMS NEW: WMS tags to allow jobs require special site/CE/queue properties CHANGES: DownloadInputData, InputDataByProtocol, InputDataResolution - allows to get multiple PFNs for the protocol resolution NEW: JobDB, JobMonitoringHandler - added traceJobParameters(s)() methods CHANGE: TaskQueueDirector - use ObjectLoader to load directors CHANGE: dirac-pilot - use Python 2.7 by default, 2014-04-09 LCG bundles *DMS NEW: DataManager to replace ReplicaManager class ( simplification, streamlining ) FIX: InputDataByProtocol - fix the case where file is only on tape FIX: FTSAgent - multiple fixes BUGFIX: ReplicateAndRegister - do not ask SE with explicit SRM2 protocol *Interfaces CHANGE: Dirac - instantiate SandboxStoreClient and WMSClient when needed, not in the constructor CHANGE: Job - removed setSystemConfig() method NEW: Job.py - added setTag() interface *Resources CHANGE: StorageElement - changes to avoid usage PFNs FIX: XROOTStorage, SRM2Storage - changes in PFN construction NEW: PoolComputingElement - a CE allowing to manage multi-core slots FIX: SSHTorqueComputingElement - specify the SSHUser user for querying running/waiting jobs *RSS NEW: added commands dirac-rss-query-db and dirac-rss-query-dtcache *RMS CHANGE: ReqDB - added Foreign Keys to ReqDB tables NEW: dirac-rms-reset-request command FIX: RequestTask - always execute operations with owner proxy *SMS FIX: few minor fixes to avoid pylint warnings [v6r10p25] *DMS CHANGE: FileCatalog - optimized file selection by metadata [v6r10p24] *DMS FIX: FC.FileMetadata - optimized queries for list interception evaluation [v6r10p23] *Resoures CHANGE: SSHComputingElement - allow SSH options to be passed from CS setup of SSH Computing Element FIX: SSHComputingElement - use SharedArea path as $HOME by default [v6r10p22] *CS CHANGE: Operations helper - if not given, determine the VO from the current proxy *Resources FIX: glexecComputingElement - allows Application Failed with Errors results to show through, rather than be masked by false "glexec CE submission" errors *DMS CHANGE: ReplicaManager - in getReplicas() rebuild PFN if /DataManagement/UseCatalogPFN option is set to False ( True by default ) [v6r10p21] *Configuration FIX: CSGlobals - allow to specify extensions in xxxDIRAC form in the CS *Interfaces FIX: Job - removed self.reqParams FIX: Job - setSubmitPools renamed to setSubmitPool, fixed parameter definition string *WMS FIX: JobMonitorigHandler, JobPolicy - allow JobMonitor property to access job information [v6r10p20] *DMS FIX: FTSAgent/Client, ReplicateAndRegister - fixes to properly process failed FTS request scheduling [v6r10p19] *DMS FIX: FTSAgent - putRequest when leaving processRequest FIX: ReplicaManager - bug in getReplicas() in dictionary creation [v6r10p18] *DMS FIX: ReplicateAndRegister - dictionary items incorrectly called in ftsTransfer() [v6r10p17] *RMS FIX: RequestDB.py - typo in a table name NEW: ReqManagerHandler - added getDistinctValues() to allow selectors in the web page *DMS CHANGE: ReplicaManager - bulk PFN lookup in getReplicas() [v6r10p16] *Framework NEW: PlottingClient - added curveGraph() function *Transformation FIX: TaskManagerAgentBase - add the missing Scheduled state *WMS FIX: TaskQueueDB - reduced number of lines in the matching parameters printout *DMS FIX: dirac-dms-show-se-status - exit on error in the service call, closes #1840 *Interface FIX: API.Job - removed special interpretation of obsoleted JDLreqt type parameters *Resources FIX: SSHComputingElement - increased timeout in getJobStatusOnHost() ssh call, closes #1830 [v6r10p15] *DMS FIX: FTSAgent - added missing monitoring activity FIX: FileCatalog - do not check directory permissions when creating / directory *Resources FIX: SSHTorqueComputingElement - removed obsoleted stuff [v6r10p14] *SMS FIX: RequestPreparationAgent - typo fixed [v6r10p13] *SMS FIX: RequestPreparationAgent - use ReplicaManager to get active replicas *DMS FIX: ReplicaManager - getReplicas returns all replicas ( in all statuses ) by default CHANGE: FC/SecurityManager - give full ACL access to the catalog to groups with admin rights *WMS CHANGE: SiteDirector - changes to reduce the load on computing elements FIX: JobWrapper - do not set Completed status for the case with failed application thread [v6r10p12] *WMS CHANGE: Replace consistently everywhere SAM JobType by Test JobType FIX: JobWrapper - the outputSandbox should be always uploaded (outsized, in failed job) *DMS FIX: RemoveFile - bugfix FIX: ReplicateAndRegister - fixes in the checksum check, retry failed FTS transfer with RM transfer NEW: RegisterReplica request operation *RMS FIX: ReqClient - fix in the request state machine FIX: Request - enhance digest string NEW: dirac-dms-reset-request command CHANGE: dirac-rms-show-request - allow selection of a request by job ID *TS FIX: TransformationDB - in getTransformationParameters() dropped "Submitted" counter in the output [v6r10p11] *Core FIX: X509Chain - cast life time to int before creating cert *Accounting FIX: DataStoreClient - self.__maxRecordsInABundle = 5000 instead of 1000 FIX: JobPolicy - allow access for JOB_MONITOR property *RMS FIX: ReqClient - fix the case when a job is Completed but in an unknown minor status *Resources BUGFIX: ProxyStorage - use checkArgumentFormat() instead of self.__checkArgumentFormatDict() [v6r10p10] *DMS FIX: Several fixes to make FTS accounting working (FTSAgent/Job, ReplicaManager, File ) [v6r10p9] *Core BUGFIX: LineGraph - Ymin was set to a minimal plot value rather than 0. *DMS CHANGE: FTSJob(Agent) - get correct information for FTS accounting (registration) [v6r10p8] *Core FIX: InstallTools - admin e-mail default location changed *Framework FIX: SystemAdministratorClientCLI - allow "set host localhost" FIX: BundleDelivery - protect against empty bundle *WMS FIX: SiteDirector - Pass siteNames and ceList as None if any is accepted FIX: WorkloadManagement.ConfigTemplate.SiteDorectory - set Site to Any by default *DMS FIX: FileCatalogCLI - ignore Datasets in ls command for backward compatibility *Resources FIX: SSH - some platforms use Password instead of password prompt [v6r10p7] *Core FIX: dirac-install - execute dirac-fix-mysql-script and dirac-external-requirements after sourcing the environment FIX: InstallTools - set basedir variable in fixMySQLScript() FIX: InstallTools - define user root@host.domain in installMySQL() *Framework BUGFIX: SystemAdministratorCLI - bug fixed in default() call signature *DMS FIX: FTSRequest - handle properly FTS server in the old system FIX: ReplicaManager - check if file is in FC before removing FIX: Request/RemovalTask - handle properly proxies for removing files BUGFIX: DatasetManager - in the table description [v6r10p6] *Core FIX: X509Certificate - reenabled fix in getDIRACGroup() *Configuration FIX: CSAPI - Group should be taken from the X509 chain and not the certificate *RMS CHANGE: ReqClient - if the job does not exist, do not try further finalization [v6r10p5] *Core FIX: X509Certificate - reverted fix in getDIRACGroup() [v6r10p4] *Core NEW: dirac-info - extra printout CHANGE: PrettyPrint - extra options in printTable() FIX: X509Certificate - bug fixed in getDIRACGroup() *Framework NEW: SystemAdministratorCLI - new showall command to show components across hosts NEW: ProxyDB - allow to upload proxies without DIRAC group *RMS CHANGE: ReqClient - requests from failed jobs update job status to Failed CHANGE: RequestTask - retry in the request finalize() [v6r10p3] *Configuration CHANGE: Registry - allow to define a default group per user *WMS BUGFIX: JobReport - typo in generateForwardDISET() [v6r10p2] *TMS CHANGE: Backward compatibility fixes when setting the Transformation files status *DMS BUGFIX: ReplicateAndRegister - bugfix when replicating to multiple destination by ReplicaManager *WMS BUGFIX: JobManager - bug fix when deleting no-existing jobs [v6r10p1] *RMS FIX: ReqDB.Operations - Arguments field changed type from BLOB to MEDIUMBLOB *DMS FIX: FileCatalog - check for non-exiting directories in removeDirectory() *TMS FIX: TransformationDB - removed constraint that was making impossible to derive a production [v6r10] *Core FIX: Several fixes on DB classes(AccountingDB, SystemLoggingDB, UserProfileDB, TransformationDB, JobDB, PilotAgentsDB) after the new movement to the new MySQL implementation with a persistent connection per running thread NEW: SystemAdministratorCLI - better support for executing remote commands FIX: DIRAC.__init__.py - avoid re-definition of platform variable NEW: Graphs - added CurveGraph class to draw non-stacked lines with markers NEW: Graphs - allow graphs with negative Y values NEW: Graphs - allow to provide errors with the data and display them in the CurveGraph FIX: InstallTools - fix for creation of the root@'host' user in MySQL FIX: dirac-install - create links to permanent directories before module installation CHANGE: InstallTools - use printTable() utility for table printing CHANGE: move printTable() utility to Core.Utilities.PrettyPrint NEW: added installation configuration examples FIX: dirac-install - fixBuildPath() operates only on files in the directory FIX: VOMSService - added X-VOMS-CSRF-GUARD to the html header to be compliant with EMI-3 servers *CS CHANGE: getVOMSVOForGroup() uses the VOMSName option of the VO definition NEW: CE2CSAgent - added ARC CE information lookup *Framework FIX: SystemAdministratorIntegrator - use Host option to get the host address in addition to the section name, closes #1628 FIX: dirac-proxy-init - uses getVOMSVOForGroup() when adding VOMS extensions *DMS CHANGE: DFC - optimization and bug fixes of the bulk file addition FIX: TransferAgent - protection against badly defined LFNs in collectFiles() NEW: DFC - added getDirectoryReplicas() service method support similar to the LFC CHANGE: DFC - added new option VisibleReplicaStatus which is used in replica getting commands CHANGE: FileCatalogClientCLI client shows number of replicas in the 2nd column rather than unimplemented number of links CHANGE: DFC - optimizations for the bulk replica look-up CHANGE: DFC updated scalability testing tool FC_Scaling_test.py NEW: DFC - methods returning replicas provide also SE definitions instead of PFNs to construct PFNs on the client side NEW: DFC - added getReplicasByMetadata() interface CHANGE: DFC - optimized getDirectoryReplicas() CHANGE: FileCatalogClient - treat the reduced output from various service queries restoring LFNs and PFNs on the fly NEW: DFC - LFNPFNConvention flag can be None, Weak or Strong to facilitate compatibility with LFC data CHANGE: FileCatalog - do not return PFNs, construct them on the client side CHANGE: FileCatalog - simplified FC_Scaling_test.py script NEW: FileCatalog/DatasetManager class to define and manipulate datasets corresponding to meta queries NEW: FileCatalogHandler - new interface methods to expose DatasetManager functionality NEW: FileCatalogClientCLI - new dataset family of commands FIX: StorageFactory, ReplicaManager - resolve SE alias name recursively FIX: FTSRequest, ReplicaManager, SRM2Storage - use current proxy owner as user name in accounting reports, closes #1602 BUGFIX: FileCatalogClientCLI - bug fix in do_ls, missing argument to addFile() call, closes #1658 NEW: FileCatalog - added new setMetadataBulk() interface, closes #1358 FIX: FileCatalog - initial argument check strips off leading lfn:, LFN:, /grid, closes #448 NEW: FileCatalog - added new setFileStatus() interface, closes #170, valid and visible file and replica statuses can be defined in respective options. CHANGE: multiple new FTS system fixes CHANGE: uniform argument checking with checkArgumentFormat() in multiple modules CHANGE: FileCatalog - add Trash to the default replica valid statuses CHANGE: ReplicaManager,FTSRequest,StorageElement - no use of PFN as returned by the FC except for file removal, rather constructing it always on the fly *SMS CHANGE: PinRequestAgent, SENamespaceCatalogCheckAgent - removed CHANGE: Use StorageManagerClient instead of StorageDB directly *WMS CHANGE: JobPolicy - optimization for bulk job verification NEW: JobPolicy - added getControlledUsers() to get users which jobs can be accessed for a given operation CHANGE: JobMonitoringHandler - Avoid doing a selection of all Jobs, first count matching jobs and then use "limit" to select only the required JobIDs. NEW: JobMonitoringHandler - use JobPolicy to filter jobs in getJobSummaryWeb() NEW: new Operations option /Services/JobMonitoring/GlobalJobsInfo ( True by default ) to allow or not job info lookup by anybody, used in JobMonitoringHandler BUGFIX: SiteDirector - take into account the target queue Platform BUGFIX: JobDB - bug in __insertNewJDL() CHANGE: dirac-admin-show-task-queues - enhanced output CHANGE: JobLoggingDB.sql - use trigger to manage the new LoggingInfo structure CHANGE: JobWrapper - trying several times to upload a request before declaring the job failed FIX: JobScheduling executor - fix race condition that causes a job to remain in Staging NEW: SiteDirector - do not touch sites for which there is no work available NEW: SiteDirector - allow sites not in mask to take jobs with JobType Test NEW: SiteDirector - allow 1 hour grace period for pilots in Unknown state before aborting them CHANGE: Allow usage of non-plural form of the job requirement options ( PilotType, GridCE, BannedSite, SubmitPool ), keep backward compatibility with a plural form *RSS FIX: DowntimeCommand - take the latest Downtime that fits NEW: porting new Policies from integration NEW: RSS SpaceToken command querying endpoints/tokens that exist *Resources NEW: added SSHOARComputingElement class NEW: added XROOTStorage class FIX: CREAMComputingElement - extra checks for validity of returned pilot references *TS CHANGE: TransformationClient(DB,Manager) - set file status for transformation as bulk operation CHANGE: TransformationClient - applying state machine when changing transformation status BUGFIX: TransformationClient(Handler) - few minor fixes NEW: TransformationDB - backported __deleteTransformationFileTask(s) methods CHANGE: TransformationDB(Client) - fixes to reestablish the FileCatalog interface FIX: TransformationAgent - added MissingInFC to consider for Removal transformations BUGFIX: TransformationAgent - in _getTransformationFiles() variable 'now' was not defined FIX: TransformationDB.sql - DataFiles primary key is changed to (FileID) from (FileID,LFN) CHANGE: TransformationDB(.sql) - schema changes suitable for InnoDB FIX: TaskManager(AgentBase) - consider only submitted tasks for updating status CHANGE: TransformationDB(.sql) - added index on LFN in DataFiles table *RMS NEW: Migrate to use the new Request Management by all the clients CHANGE: RequestContainer - Retry failed transfers 10 times and avoid sub-requests to be set Done when the files are failed CHANGE: Use a unique name for storing the proxy as processes may use the same "random" name and give conflicts NEW: RequestClient(Handler) - add new method readRequest( requestname) *Workflow NEW: Porting the LHCb Workflow package to DIRAC to make the use of general purpose modules and simplify construction of workflows [v6r9p33] *Accounting BUGFIX: AccountingDB - wrong indentation [v6r9p32] *Accounting FIX: AccountingDB - use old style grouping if the default grouping is altered, e.g. by Country [v6r9p31] *Accounting CHANGE: AccountingDB - changes to speed up queries: use "values" in GROUP By clause; drop duplicate indexes; reorder fields in the UniqueConstraint index of the "bucket" tables [v6r9p30] *DMS CHANGE: FileCatalogFactory - construct CatalogURL from CatalogType by default *SMS FIX: dirac-stager-stage-files - changed the order of the arguments [v6r9p29] *TS FIX: TaskManager(AgentBase) - fix for considering only submitted tasks [v6r9p28] *TS FIX: TransformationDB(ManagerHandler) - several portings from v6r10 [v6r9p27] *SMS FIX: StorageManagementDB - in removeUnlinkedReplicas() second look for CacheReplicas for which there is no entry in StageRequests [v6r9p26] *Resources CHANGE: CREAMComputigElement - Make sure that pilots submitted to CREAM get a fresh proxy during their complete lifetime *Framework FIX: ProxyDB - process properly any SQLi with DNs/groups with 's in the name [v6r9p25] *TS CHANGE: TransformationClient - changed default timeout values for service calls FIX: TransformationClient - fixes for processing of derived transformations [v6r9p24] *TS FIX: TransformationClient - in moveFilesToDerivedTransformation() set file status to Moved- [v6r9p23] *Core BUGFIX: InstallTools - improper configuration prevents a fresh new installation *WMS BUGFIX: PilotDirector - Operations Helper non-instantiated [v6r9p22] *WMS FIX: PilotDirector - allow to properly define extensions to be installed by the Pilot differently to those installed at the server FIX: Watchdog - convert pid to string in ProcessMonitor *TS FIX: TransformationDB - splitting files in chunks *DMS NEW: dirac-dms-create-removal-request command CHANGE: update dirac-dms-xxx commands to use the new RMS client, strip lines when reading LFNs from a file [v6r9p21] *TS FIX: Transformation(Client,DB,Manager) - restored FileCatalog compliant interface FIX: TransformationDB - fix in __insertIntoExistingTransformationFiles() [v6r9p20] *Core BUGFIX: ProxyUpload - an on the fly upload does not require a proxy to exist *DMS CHANGE: TransferAgent - use compareAdler() for checking checksum FIX: FailoverTransfer - recording the sourceSE in case of failover transfer request *WMS FIX: ProcessMonitor - some fixes added, printout when <1 s of consumed CPU is found *Transformation BUGFIX: TransformationClient - fixed return value in moveFilesToDerivedTransformation() *RMS BUGFIX: CleanReqDBAgent - now() -> utcnow() in initialize() *Resources FIX: ARCComputingElement - fix the parsing of CE status if no jobs are available [v6r9p19] *DMS FIX: FileCatalog/DirectoryMetadata - inherited metadata is used while selecting directories in findDirIDsByMetadata() [v6r9p18] *DMS FIX: FTSSubmitAgent, FTSRequest - fixes the staging mechanism in the FTS transfer submission NEW: TransferDBMonitoringHandler - added getFilesForChannel(), resetFileChannelStatus() [v6r9p17] *Accounting FIX: DataStoreClient - send accounting records in batches of 1000 records instead of 100 *DMS: FIX: FailoverTransfer - catalog name from list to string FIX: FTSSubmitAgent, FTSRequest - handle FTS3 as new protocol and fix bad submission time FIX: FTSSubmitAgent, FTSRequest - do not submit FTS transfers for staging files *WMS FIX: TaskQueueDB - do not check enabled when TQs are requested from Directors FIX: TaskQueueDB - check for Enabled in the TaskQueues when inserting jobs to print an alert NEW: TaskQueueDB - each TQ can have at most 5k jobs, if beyond the limit create a new TQ to prevent long matching times when there are way too many jobs in a single TQ [v6r9p16] *TS BUGFIX: typos in TransformationCleaningAgent.py *DMS CHANGE: DownloadInputData - check the available disk space in the right input data directory FIX: DownloadInputData - try to download only Cached replicas [v6r9p15] *Core FIX: MySQL - do not decrease the retry counter after ping failure *DMS CHANGE: FC/DirectoryMetadata - Speed up findFilesByMetadataWeb when many files match FIX: RemovalTask - fix error string when removing a non existing file (was incompatible with the LHCb BK client). *WMS FIX: JobReport - minor fix ( removed unused imports ) FIX: JobMonitoring(JobStateUpdate)Handler - jobID argument can be either string, int or long *TS CHANGE: TransformationClient - change status of Moved files to a deterministic value FIX: FileReport - minor fix ( inherits object ) [v6r9p14] *DMS CHANGE: FTSDB - changed schema: removing FTSSite table. From now on FTS sites would be read from CS Resources [v6r9p13] FIX: included fixes from v6r8p26 patch release [v6r9p12] FIX: included fixes from v6r8p25 patch release [v6r9p11] *DMS BUGFIX: FTSRequest - in __resolveFTSServer() type "=" -> "==" [v6r9p10] FIX: included fixes from v6r8p24 patch release *Core NEW: StateMachine utility *DMS BUGFIX: in RegisterFile operation handler *Interfaces FIX: Dirac.py - in splitInputData() consider only Active replicas [v6r9p9] *RMS FIX: RequestDB - added getRequestFileStatus(), getRequestName() methods [v6r9p8] *DMS FIX: RequestDB - get correct digest ( short request description ) of a request [v6r9p7] FIX: included fixes from v6r8p23 patch release *RSS FIX: SpaceTokenOccupancyPolicy - SpaceToken Policy decision was based on percentage by mistake *RMS NEW: new scripts dirac-dms-ftsdb-summary, dirac-dms-show-ftsjobs FIX: FTSAgent - setting space tokens for newly created FTSJobs [v6r9p6] *DMS BUGFIX: dirac-admin-add-ftssite - missing import *RMS NEW: RequestDB, ReqManagerHandler - added getRequestStatus() method *TS FIX: fixes when using new RequestClient with the TransformationCleaningAgent *WMS BUGFIX: typo in SandboxStoreHandler transfer_fromClient() method [v6r9p5] *DMS BUGFIX: missing proxy in service env in the FTSManager service. By default service will use DataManager proxy refreshed every 6 hours. *Resources NEW: StorageElement - new checkAccess policy: split the self.checkMethods in self.okMethods. okMethods are the methods that do not use the physical SE. The isValid returns S_OK for all those immediately *RSS FIX: SpaceTokenOccupancyPolicy - Policy that now takes into account absolute values for the space left *TS FIX: TransformationCleaningAgent - will look for both old and new RMS [v6r9p4] *Stager NEW: Stager API: dirac-stager-monitor-file, dirac-stager-monitor-jobs, dirac-stager-monitor-requests, dirac-stager-show-stats [v6r9p3] *Transformation FIX: TransformationCleaning Agent status was set to 'Deleted' instead of 'Cleaned' [v6r9p2] *RSS NEW: Added Component family tables and statuses FIX: removed old & unused code NEW: allow RSS policies match wild cards on CS *WMS BUGFIX: FailoverTransfer,JobWrapper - proper propagation of file metadata [v6r9p1] *RMS NEW: FTSAgent - update rwAccessValidStamp, update ftsGraphValidStamp, new option for staging files before submission, better log handling here and there CHANGE: FTSJob - add staging flag in in submitFTS2 CHANGE: Changes in WMS (FailoverTransfer, JobReport, JobWrapper, SandboxStoreHandler) and TS (FileReport) to follow the new RMS. NEW: Full CRUD support in RMS. *RSS NEW: ResourceManagementDB - new table ErrorReportBuffer NEW: new ResourceManagementClient methods - insertErrorReportBuffer, selectErrorReportBuffer, deleteErrorReportBuffer [v6r9] NEW: Refactored Request Management System, related DMS agents and FTS management components [v6r8p28] *Core BUGFIX: RequestHandler - the lock Name includes ActionType/Action *DMS FIX: dirac-dms-filecatalog-cli - prevent exception in case of missing proxy [v6r8p27] *DMS BUGFIX: dirac-dms-add-file - fixed typo item -> items [v6r8p26] *Core NEW: RequestHandler - added getServiceOption() to properly resolve inherited options in the global service handler initialize method NEW: FileCatalogHandler, StorageElementHandler - use getServiceOption() [v6r8p25] FIX: included fixes from v6r7p40 patch release *Resources FIX: SRM2Storage - do not account gfal_ls operations [v6r8p24] FIX: included fixes from v6r7p39 patch release *Core FIX: SiteSEMapping was returning wrong info *DMS FIX: FTSRequest - choose explicitly target FTS point for RAL and CERN BUGFIX: StrategyHandler - wrong return value in __getRWAccessForSE() *Resources CHANGE: SRM2Storage - do not account gfal_ls operations any more [v6r8p23] FIX: included fixes from v6r7p37 patch release *TS FIX: TransformationDB - allow tasks made with ProbInFC files FIX: TransformationCleaingAgent,Client - correct setting of transformation status while cleaning [v6r8p22] FIX: included fixes from v6r7p36 patch release [v6r8p21] *DMS FIX: FileCatalog/DirectoryMetadata - even if there is no meta Selection the path should be considered when getting Compatible Metadata FIX: FileCatalog/DirectoryNodeTree - findDir will return S_OK( '' ) if dir not found, always return the same error from DirectoryMetadata in this case. *RSS FIX: DowntimeCommand - use UTC time stamps *TS FIX: TransformationAgent - in _getTransformationFiles() get also ProbInFC files in addition to Used [v6r8p20] *Stager NEW: Stager API: dirac-stager-monitor-file, dirac-stager-monitor-jobs, dirac-stager-monitor-requests, dirac-stager-show-stats [v6r8p19] *Transformation FIX: TransformationCleaning Agent status was set to 'Deleted' instead of 'Cleaned' [v6r8p18] *TS BUGFIX: TransformationAgent - regression in __cleanCache() [v6r8p17] FIX: included fixes from v6r7p32 patch release *WMS FIX: StalledJobAgent - for accidentally stopped jobs ExecTime can be not set, set it to CPUTime for the accounting purposes in this case [v6r8p16] FIX: included fixes from v6r7p31 patch release *WMS BUGFIX: TaskQueueDB - fixed a bug in the negative matching conditions SQL construction *RSS NEW: improved doc strings of PEP, PDP modules ( part of PolicySystem ) FIX: Minor changes to ensure consistency if ElementInspectorAgent and users interact simultaneously with the same element CHANGE: removed DatabaseCleanerAgent ( to be uninstalled if already installed ) FIX: SummarizeLogsAgent - the logic of the agent was wrong, the agent has been re-written. [v6r8p15] *Core FIX: X509Chain - fix invalid information when doing dirac-proxy-info without CS ( in getCredentials() ) *RSS NEW: PDP, PEP - added support for option "doNotCombineResult" on PDP [v6r8p14] *Core FIX: dirac-deploy-scripts - can now work with the system python *WMS NEW: dirac-wms-cpu-normalization - added -R option to modify a given configuration file FIX: Executor/InputData - Add extra check for LFns in InputData optimizer, closes #1472 *Transformation CHANGE: TransformationAgent - add possibility to kick a transformation (not skip it if no unused files), by touching a file in workDirectory BUGFIX: TransformationAgent - bug in __cleanCache() dict modified in a loop [v6r8p13] *Transformation BUGFIX: TransformationDB - restored import of StringType [v6r8p12] NEW: Applied patches from v6r7p29 *WMS FIX: JobDB - check if SystemConfig is present in the job definition and convert it into Platform *DMS FIX: ReplicaManager - do not get metadata of files when getting files in a directory if not strictly necessary *RSS NEW: ported from LHCb PublisherHandler for RSS web views [v6r8p11] NEW: Applied patches from v6r7p27 *RSS NEW: SpaceTokenOccupancyPolicy - ported from LHCbDIRAC NEW: db._checkTable done on service initialization ( removed dirac-rss-setup script doing it ) *Transformation FIX: TaskManager - reset oJob for each task in prepareTransformationTasks() BUGFIX: ValidateOutputDataAgent - typo fixed in getTransformationDirectories() FIX: TransformationManagerHandler - use CS to get files statuses not to include in processed file fraction calculation for the web monitoring pages [v6r8p10] NEW: Applied patches from v6r7p27 [v6r8p9] *DMS FIX: TransferAgent,dirac-dms-show-se-status, ResourceStatus,TaskManager - fixes needed for DMS components to use RSS status information NEW: ReplicaManager - allow to get metadata for an LFN+SE as well as PFN+SE [v6r8p8] *RSS BUGFIX: dirac-rss-setup - added missing return of S_OK() result [v6r8p7] NEW: Applied patches from v6r7p24 *DMS BUGFIX: LcgFileCatalogClient - bug in addFile() *RSS BUGFIX: fixed script dirac-rss-set-token, broken in the current release. NEW: Statistics module - will be used in the future to provide detailed information from the History of the elements [v6r8p6] NEW: Applied patches from v6r7p23 *Transformation FIX: TaskManager - allow prepareTransformationTasks to proceed if no OutputDataModule is defined FIX: TransformationDB - remove INDEX(TaskID) from TransformationTasks. It produces a single counter for the whole table instead of one per TransformationID *WMS FIX: WMSUtilities - to allow support for EMI UI's for pilot submission we drop support for glite 3.1 [v6r8p5] NEW: Applied patches from v6r7p22 *RSS CHANGE: removed old tests and commented out files *WMS FIX: PoolXMLCatalog - proper addFile usage *Transformation CHANGE: TransformationAgent - clear replica cache when flushing or setting a file in the workdirectory [v6r8p4] *Transformation FIX: The connection to the jobManager is done only at submission time FIX: Jenkins complaints fixes *WMS BUGFIX: JobDB - CPUtime -> CPUTime FIX: Jenkins complaints fixes [v6r8p3] *DMS BUGFIX: LcgFileCatalogClient [v6r8p2] *DMS: FIX: LcgFileCatalogClient - remove check for opening a session in __init__ as credentials are not yet set *Transformation CHANGE: reuse RPC clients in Transformation System [v6r8p1] *Core FIX: dirac-deploy-scripts - restored regression w.r.t. support of scripts starting with "d" *DMS BUGFIX: LcgFileCatalogClient - two typos fixed [v6r8] CHANGE: Several fixes backported from the v7r0 integration branch *Core CHANGE: DictCache - uses global LockRing to avoid locks in multiprocessing FIX: X509Chain - proxy-info showing an error when there's no CS *DMS FIX: TransferAgent - inside loop filter out waiting files dictionary BUGFIX: dirac-admin-allow-se - there was a continue that was skipping the complete loop for ARCHIVE elements NEW: LcgFileCatalogClient - test return code in startsess lfc calls *WMS: FIX: OptimizerExecutor, InputData, JobScheduling - check that site candidates have all the replicas *RSS: BUGFIX: ResourceStatus, RSSCacheNoThread - ensure that locks are always released *Transformation FIX: TaskManager - site in the job definition is taken into account when submitting NEW: Transformation - get the allowed plugins from the CS /Operations/Transformations/AllowedPlugins FIX: ValidateOutputDataAgent - self not needed for static methods [v6r7p40] *Resources FIX: StorageElement class was not properly passing the lifetime argument for prestageFile method [v6r7p39] *Core CHANGE: Grid - in executeGridCommand() allow environment script with arguments needed for ARC client *DMS FIX: DFC SEManager - DIP Storage can have a list of ports now *Resources FIX: ARCComputingElement - few fixes after debugging [v6r7p38] *Core NEW: DISET FileHelper, TransferClient - possibility to switch off check sum *Resources NEW: ARCComputingElement - first version NEW: StorageFactory - possibility to pass extra protocol parameters to storage object NEW: DIPStorage - added CheckSum configuration option BUGFIX: SSHComputingElement - use CE name in the pilot reference construction *WMS FIX: StalledJobAgent - if ExecTime < CPUTime make it equal to CPUTime [v6r7p37] *Framework BUGFIX: NotificationDB - typos in SQL statement in purgeExpiredNotifications() *WMS NEW: JobCleaningAgent - added scheduling sandbox LFN removal request when deleting jobs CHANGE: JobWrapper - report only error code as ApplicationError parameter when payload finishes with errors NEW: SiteDirector - possibility to specify extensions to be installed in pilots in /Operations/Pilots/Extensions option in order not to install all the server side extensions *DMS CHANGE: FileCatalogFactory - use service path as default URL CHANGE: FileCatalogFactory - use ObjectLoader to import catalog clients *SMS BUGFIX: StorageManagementDB, dirac-stager-monitor-jobs - small bug fixes ( sic, Daniela ) *Resources CHANGE: DIPStorage - added possibility to specify a list of ports for multiple service end-points CHANGE: InProcessComputingElement - demote log message when payload failure to warning, the job will fail anyway FIX: StalledJobAgent - if pilot reference is not registered, this is not an error of the StalledJobAgent, no log.error() in this case *RMS CHANGE: RequestTask - ensure that tasks are executed with user credentials even with respect to queries to DIRAC services ( useServerCertificate flag set to false ) [v6r7p36] *WMS FIX: CREAMCE, SiteDirector - make sure that the tmp executable is removed CHANGE: JobWrapper - remove sending mails via Notification Service in case of job rescheduling *SMS FIX: StorageManagementDB - fix a race condition when old tasks are set failed between stage submission and update. [v6r7p35] *Stager NEW: Stager API: dirac-stager-monitor-file, dirac-stager-monitor-jobs, dirac-stager-monitor-requests, dirac-stager-show-stats [v6r7p34] *Transformation FIX: TransformationCleaning Agent status was set to 'Deleted' instead of 'Cleaned' [v6r7p33] *Interfaces FIX: Job.py - in setExecutable() - prevent changing the log file name string type *StorageManagement NEW: StorageManagementDB(Handler) - kill staging requests at the same time as killing related jobs, closes #1510 FIX: StorageManagementDB - demote the level of several log messages [v6r7p32] *DMS FIX: StorageElementHandler - do not use getDiskSpace utility, use os.statvfs instead CHANGE: StorageManagementDB - in getStageRequests() make MySQL do an UNIQUE selection and use implicit loop to speed up queries for large results *Resources FIX: lsfce remote script - use re.search instead of re.match in submitJob() to cope with multipline output [v6r7p31] *WMS FIX: SiteDirector - make possible more than one SiteDirector (with different pilot identity) attached to a CE, ie sgm and pilot roles. Otherwise one is declaring Aborted the pilots from the other. [v6r7p30] *Core CHANGE: X509Chain - added groupProperties field to the getCredentials() report BUGFIX: InstallTools - in getSetupComponents() typo fixed: agent -> executor [v6r7p29] *DMS CHANGE: FileCatalog - selection metadata is also returned as compatible metadata in the result of getCompatibleMetadata() call NEW: FileCatalog - added path argument to getCompatibleMetadata() call NEW: FileCatalogClient - added getFileUserMetadata() BUGFIX: dirac-dms-fts-monitor - exit with code -1 in case of error *Resources FIX: CREAMComputingElement - check globus-url-copy result for errors when retrieving job output [v6r7p28] *DMS BUGFIX: FileCatalog/DirectoryMetadata - wrong MySQL syntax [v6r7p27] *Core FIX: Mail.py - fix of the problem of colons in the mail's body *Interfaces NEW: Job API - added setSubmitPools(), setPlatform() sets ... "Platform" *WMS FIX: TaskQueueDB - use SystemConfig as Platform for matching ( if Platform is not set explicitly *Resources FIX: SSHComputingElement - use ssh host ( and not CE name ) in the pilot reference BUGFIX: SSHGEComputingElement - forgotten return statement in _getJobOutputFiles() *Framework NEW: dirac-sys-sendmail - email's body can be taken from pipe. Command's argument in this case will be interpreted as a destination address [v6r7p26] *DMS FIX: ReplicaManager - status names Read/Write -> ReadAccess/WriteAccess [v6r7p25] *Core CHANGE: X509Chain - in getCredentials() failure to contact CS is not fatal, can happen when calling dirac-proxy-init -x, for example [v6r7p24] *DMS NEW: FileCatalog - added getFilesByMetadataWeb() to allow pagination in the Web catalog browser *WMS CHANGE: WMSAdministrator, DiracAdmin - get banned sites list by specifying the status to the respective jobDB call [v6r7p23] *Transformation BUGFIX: TransformationDB - badly formatted error log message *RMS CHANGE: RequestDBMySQL - speedup the lookup of requests *WMS BUGFIX: dirac-dms-job-delete - in job selection by group *DMS FIX: LcgFileCatalogClient - getDirectorySize made compatible with DFC BUGFIX: LcgFileCatalogClient - proper call of __getClientCertInfo() [v6r7p22] *Transformation CHANGE: InputDataAgent - treats only suitable transformations, e.g. not the extendable ones. CHANGE: TransformationAgent - make some methods more public for easy overload [v6r7p21] *Core FIX: Shifter - pass filePath argument when downloading proxy [v6r7p20] *DMS CHANGE: StrategyHandler - move out SourceSE checking to TransferAgent CHANGE: ReplicaManager, InputDataAgent - get active replicas FIX: StorageElement, SRM2Storage - support for 'xxxAccess' statuses, checking results of return structures *RSS NEW: set configurable email address on the CS to send the RSS emails NEW: RSSCache without thread in background FIX: Synchronizer - moved to ResourceManager handler [v6r7p19] *DMS BUGFIX: ReplicaManager - in putAndRegister() SE.putFile() singleFile argument not used explicitly [v6r7p18] *WMS FIX: StalledJobAgent - do not exit the loop over Completed jobs if accounting sending fails NEW: dirac-wms-job-delete - allow to specify jobs to delete by job group and/or in a file FIX: JobManifest - If CPUTime is not set, set it to MaxCPUTime value [v6r7p17] *Resources FIX: SRM2Storage - treat properly "22 SRM_REQUEST_QUEUED" result code [v6r7p16] *DMS FIX: StrategyHandler - do not proceed when the source SE is not valid for read BUGFIX: StorageElement - putFile can take an optional sourceSize argument BUGFIX: ReplicaManager - in removeFile() proper loop on failed replicas *RSS FIX: SpaceTokenOccupancyCommand, CacheFeederAgent - add timeout when calling lcg_util commands *WMS FIX: JobManifest - take all the SubmitPools defined in the TaskQueueAgent NEW: StalledJobAgent - declare jobs stuck in Completed status as Failed [v6r7p15] *Core BUGFIX: SocketInfo - in host identity evaluation *DMS BUGFIX: FileCatalogHandler - missing import os *Transformation CHANGE: JobManifest - getting allowed job types from operations() section [v6r7p14] *DMS CHANGE: StorageElementProxy - removed getParameters(), closes #1280 FIX: StorageElementProxy - free the getFile space before the next file FIX: StorageElement - added getPFNBase() to comply with the interface *Interfaces CHANGE: Dirac API - allow lists of LFNs in removeFile() and removeReplica() *WMS CHANGE: JobSchedulingAgent(Executor) - allow both BannedSite and BannedSites JDL option *RSS FIX: ElementInspectorAgent - should only pick elements with rss token ( rs_svc ). FIX: TokenAgent - using 4th element instead of the 5th. Added option to set admin email on the CS. [v6r7p13] *Core FIX: Resources - in getStorageElementSiteMapping() return only sites with non-empty list of SEs *DMS FIX: StorageElement - restored the dropped logic of using proxy SEs FIX: FileCatalog - fix the UseProxy /LocalSite/Catalog option *Transformation FIX: TransformationDB - use lower() string comparison in extendTransformation() [v6r7p12] *WMS BUGFIX: JobManifest - get AllowedSubmitPools from the /Systems section, not from /Operations *Core NEW: Resources helper - added getSites(), getStorageElementSiteMapping() *DMS CHANGE: StrategyHandler - use getStorageElementSiteMapping helper function BUGFIX: ReplicaManager - do not modify the loop dictionary inside the loop [v6r7p11] *Core CHANGE: Subprocess - put the use of watchdog in flagging [v6r7p10] *Core NEW: Logger - added getLevel() method, closes #1292 FIX: Subprocess - returns correct structure in case of timeout, closes #1295, #1294 CHANGE: TimeOutExec - dropped unused utility FIX: Logger - cleaned unused imports *RSS CHANGE: ElementInspectorAgent - do not use mangled name and removed shifterProxy agentOption [v6r7p9] *Core BUGFIX: InstallTools - MySQL Port should be an integer [v6r7p8] *Core FIX: Subprocess - consistent timeout error message *DMS NEW: RemovalTask - added bulk removal FIX: StrategyHandler - check file source CEs CHANGE: DataIntegrityClient - code beautification CHANGE: ReplicaManager - do not check file existence if replica information is queried anyway, do not fail if file to be removed does not exist already. [v6r7p7] FIX: Several fixes to allow automatic code documentation *Core NEW: InstallTools - added mysqlPort and mysqlRootUser *DMS CHANGE: ReplicaManager - set possibility to force the deletion of non existing files CHANGE: StrategyHandler - better handling of checksum check during scheduling [v6r7p6] *Core FIX: dirac-install - restore signal alarm if downloadable file is not found FIX: Subprocess - using Manager proxy object to pass results from the working process *DMS: CHANGE: StorageElement - removed overwride mode CHANGE: removed obsoleted dirac-dms-remove-lfn-replica, dirac-dms-remove-lfn NEW: FTSMonitorAgent - filter out sources with checksum mismatch FIX: FTSMonitorAgent, TransferAgent - fix the names of the RSS states *RSS NEW: ElementInspectorAgent runs with a variable number of threads which are automatically adjusted NEW: Added policies to force a particular state, can be very convenient to keep something Banned for example. NEW: policy system upgrade, added finer granularity when setting policies and actions *WMS NEW: SiteDirector- allow to define pilot DN/Group in the agent options CHANGE: JobDescription, JobManifest - take values for job parameter verification from Operations CS section [v6r7p5] *Interfaces BUGFIX: dirac-wms-job-get-output - properly treat the case when output directory is not specified [v6r7p4] *Core FIX: Subprocess - avoid that watchdog kills the executor process before it returns itself *Framework BUGFIX: ProxuManagerClient - wrong time for caching proxies *RSS FIX: removed obsoleted methods *DMS NEW: FileCatalog - added findFilesByMetadataDetailed - provides detailed metadata for selected files [v6r7p3] *DMS FIX: FTSMonitorAgent - logging less verbose *Transformation FIX: TransformationAgent - use the new CS defaults locations FIX: Proper agent initialization NEW: TransformationPlaugin - in Broadcast plugin added file groupings by number of files, make the TargetSE always defined, even if the SourceSE list contains it *ResourceStatus FIX: Added the shifter's proxy to several agents *RMS FIX: RequestContainer - the execution order was not properly set for the single files *Framework: BUGFIX: ProxyManagerClient - proxy time can not be shorter than what was requested [v6r7p2] *Core FIX: dirac-configure - switch to use CS before checking proxy info *Framework NEW: dirac-sys-sendmail new command NEW: SystemAdmininistratorCLI - added show host, uninstall, revert commands NEW: SystemAdmininistratorHandler - added more info in getHostInfo() NEW: SystemAdmininistratorHandler - added revertSoftware() interface *Transformation FIX: TransformationCleaningAgent - check the status of returned results [v6r7p1] *Core FIX: Subprocess - finalize the Watchdog closing internal connections after a command execution CHANGE: add timeout for py(shell,system)Call calls where appropriate CHANGE: Shifter - use gProxyManager in a way that allows proxy caching *Framework NEW: ProxyManagerClient - allow to specify validity and caching time separately FIX: ProxyDB - replace instead of delete+insert proxy in __storeVOMSProxy *DMS NEW: FTSMonitorAgent - made multithreaded for better efficiency FIX: dirac-dms-add-file - allow LFN: prefix for lfn argument *WMS NEW: dirac-wms-job-get-output, dirac-wms-job-status - allow to retrieve output for a job group FIX: TaskQueueDB - fixed selection SQL in __generateTQMatchSQL() CHANGE: OptimizerExecutor - reduce diversity of MinorStatuses for failed executors *Resources FIX: CREAMComputingElement - remove temporary JDL right after the submission [v6r6p21] *DMS BUGFIX: TransformationCleaningAgent - use the right signature of cleanMetadataCatalogFiles() call [v6r6p20] *DMS FIX: RegistrationTask - properly escaped error messages BUGFIX: DirectoryMetadata - use getFileMetadataFields from FileMetadata in addMetadataField() NEW: When there is a missing source error spotted during FTS transfer, file should be reset and rescheduled again until maxAttempt (set to 100) is reached *WMS FIX: JobScheduling - fix the site group logic in case of Tier0 [v6r6p19] *DMS BUGFIX: All DMS agents - set up agent name in the initialization *Core NEW: Subprocess - timeout wrapper for subprocess calls BUGFIX: Time - proper interpreting of 0's instead of None CHANGE: DISET - use cStringIO for ANY read that's longer than 16k (speed improvement) + Less mem when writing data to the net FIX: Os.py - protection against failed "df" command execution NEW: dirac-info prints lcg bindings versions CHANGE: PlotBase - made a new style class NEW: Subprocess - added debug level log message *Framework NEW: SystemAdministratorIntegrator client for collecting info from several hosts NEW: SystemAdministrator - added getHostInfo() FIX: dirac-proxy-init - always check for errors in S_OK/ERROR returned structures CHANGE: Do not accept VOMS proxies when uploading a proxy to the proxy manager *Configuration FIX: CE2CSAgent - get a fresh copy of the cs data before attempting to modify it, closes #1151 FIX: Do not create useless backups due to slaves connecting and disconnecting FIX: Refresher - prevent retrying with 'Insane environment' *Accounting NEW: Accounting/Job - added validation of reported values to cope with the weird Yandex case FIX: DBUtils - take into account invalid values, closes #949 *DMS FIX: FTSSubmitAgent - file for some reason rejected from submission should stay in 'Waiting' in TransferDB.Channel table FIX: FTSRequest - fix in the log printout CHANGE: dirac-dms-add-file removed, dirac-dms-add-files renamed to dirac-dms-add-file FIX: FileCatalogCLI - check the result of removeFile call FIX: LcgFileCatalogClient - get rid of LHCb specific VO evaluation NEW: New FileCatalogProxy service - a generalization of a deprecated LcgFileCatalog service FIX: Restored StorageElementProxy functionality CHANGE: dirac-dms-add-file - added printout NEW: FileCatalog(Factory), StorageElement(Factory) - UseProxy flag moved to /Operations and /LocalSite sections *RSS NEW: general reimplementation: New DB schema using python definition of tables, having three big blocks: Site, Resource and Node. MySQLMonkey functionality almost fully covered by DB module, eventually will disappear. Services updated to use new database. Clients updated to use new database. Synchronizer updated to fill the new database. When helpers will be ready, it will need an update. One ElementInspectorAgent, configurable now is hardcoded. New Generic StateMachine using OOP. Commands and Policies simplified. ResourceStatus using internal cache, needs to be tested with real load. Fixes for the state machine Replaced Bad with Degraded status ( outside RSS ). Added "Access" to Read|Write|Check|Remove SE statuses wherever it applies. ResourceStatus returns by default "Active" instead of "Allowed" for CS calls. Caching parameters are defined in the CS FIX: dirac-admin-allow/ban-se - allow a SE on Degraded ( Degraded->Active ) and ban a SE on Probing ( Probing -> Banned ). In practice, Active and Degraded are "usable" states anyway. *WMS FIX: OptimizerExecutor - failed optimizations will still update the job NEW: JobWrapper - added LFNUserPrefix VO specific Operations option used for building user LFNs CHANGE: JobDB - do not interpret SystemConfig in the WMS/JobDB CHANGE: JobDB - Use CPUTime JDL only, keep MaxCPUTime for backward compatibility CHANGE: JobWrapper - use CPUTime job parameter instead of MaxCPUTime CHANGE: JobAgent - use CEType option instead of CEUniqueID FIX: JobWrapper - do not attempt to untar directories before having checked if they are tarfiles NEW: dirac-wms-job-status - get job statuses for jobs in a given job group *SMS FIX: StorageManagementDB - when removing unlinked replicas, take into account the case where a staging request had been submitted, but failed *Resources NEW: glexecCE - add new possible locations of the glexec binary: OSG specific stuff and in last resort looking in the PATH NEW: LcgFileCatalogClient - in removeReplica() get the needed PFN inside instead of providing it as an argument *TS CHANGE: Transformation types definition are moved to the Operations CS section *Interfaces FIX: Dirac.py - CS option Scratchdir was in LocalSite/LocalSite FIX: Dirac.py - do not define default catalog, use FileCatalog utility instead [v6r6p19] *DMS BUGFIX: All DMS agents - set up agent name in the initialization [v6r6p18] *Transformation CHANGE: /DIRAC/VOPolicy/OutputDataModule option moved to /Transformations/OutputDataModule *Resources FIX: ComputingElement - properly check if the pilot proxy has VOMS before adding it to the payload when updating it *WMS BUGFIX: JobSanity - fixed misspelled method call SetParam -> SetParameter [v6r6p17] *Transformation BUGFIX: TransformationAgent - corrected __getDataReplicasRM() [v6r6p16] *DMS FIX: Agents - proper __init__ implementation with arguments passing to the super class FIX: LcgFileCatalogClient - in removeReplica() reload PFN in case it has changed [v6r6p15] *Framework BUGFIX: ErrorMessageMonitor - corrected updateFields call *DMS: NEW: FTSMonitorAgent completely rewritten in a multithreaded way *Transformation FIX: InputDataAgent - proper instantiation of TransformationClient CHANGE: Transformation - several log message promoted from info to notice level [v6r6p14] *Transformation FIX: Correct instantiation of agents inside several scripts CHANGE: TransformationCleaningAgent - added verbosity to logs CHANGE: TransformationAgent - missingLFC to MissingInFC as it could be the DFC as well FIX: TransformationAgent - return an entry for all LFNs in __getDataReplicasRM *DMS FIX: TransferAgent - fix exception reason in registerFiles() [v6r6p13] *DMS CHANGE: TransferAgent - change RM call from getCatalogueReplicas to getActiveReplicas. Lowering log printouts here and there [v6r6p12] *DMS BUGFIX: RemovalTask - Replacing "'" by "" in error str set as attribute for a subRequest file. Without that request cannot be updated when some nasty error occurs. [v6r6p11] *RMS: BUGFIX: RequestClient - log string formatting *DMS BUGFIX: RemovalTask - handling for files not existing in the catalogue *Transformation FIX: TransformationManager - ignore files in NotProcessed status to get the % of processed files *Interfaces FIX: Fixes due to the recent changes in PromptUser utility [v6r6p10] *RMS FIX: RequestDBMySQL - better escaping of queries *WMS FIX: SiteDirector - get compatible platforms before checking Task Queues for a site [v6r6p9] *Core FIX: Utilities/PromptUser.py - better user prompt *Accounting NEW: Add some validation to the job records because of weird data coming from YANDEX.ru *DMS BUGFIX: ReplicaManager - typo errStr -> infoStr in __replicate() FIX: FTSRequest - fixed log message *WMS FIX: SiteDirector - use CSGlobals.getVO() call instead of explicit CS option [v6r6p8] *Transformation BUGFIX: TransformationDB - typo in getTransformationFiles(): iterValues -> itervalues [v6r6p7] *Resources FIX: StorageFactory - uncommented line that was preventing the status to be returned BUGFIX: CE remote scripts - should return status and not call exit() BUGFIX: SSHComputingElement - wrong pilot ID reference [v6r6p6] *WMS FIX: TaskQueueDB - in findOrphanJobs() retrieve orphaned jobs as list of ints instead of list of tuples FIX: OptimizerExecutor - added import of datetime to cope with the old style optimizer parameters *Transformation FIX: TransformationAgent - fix finalization entering in an infinite loop NEW: TransformationCLI - added resetProcessedFile command FIX: TransformationCleaningAgent - treating the archiving delay FIX: TransformationDB - fix in getTransformationFiles() in case of empty file list [v6r6p5] *Transformation FIX: TransformationAgent - type( transClient -> transfClient ) FIX: TransformationAgent - self._logInfo -> self.log.info FIX: TransformationAgent - skip if no Unused files FIX: TransformationAgent - Use CS option for replica cache lifetime CHANGE: TransformationAgent - accept No new Unused files every [6] hours [v6r6p4] *DMS FIX: TransferAgent - protection for files that can not be scheduled BUGFIX: TransferDB - typo (instIDList - > idList ) fixed *Transformation BUGFIX: TransformationAgent - typo ( loginfo -> logInfo ) [v6r6p3] FIX: merged in patch v6r5p14 *Core BUGFIX: X509Chain - return the right structure in getCredentials() in case of failure FIX: dirac-deploy-scripts.py - allow short scripts starting from "d" FIX: dirac-deploy-scripts.py - added DCOMMANDS_PPID env variable in the script wrapper FIX: ExecutorReactor - reduced error message dropping redundant Task ID *Interfaces BUGFIX: Dirac.py - allow to pass LFN list to replicateFile() *DMS FIX: FileManager - extra check if all files are available in _findFiles() BUGFIX: FileCatalogClientCLI - bug in DirectoryListing [v6r6p2] FIX: merged in patch v6r5p13 *WMS FIX: SiteDirector - if no community set, look for DIRAC/VirtualOrganization setting *Framework FIX: SystemLoggingDB - LogLevel made VARCHAR in the MessageRepository table FIX: Logging - several log messages are split in fixed and variable parts FIX: SystemLoggingDB - in insertMessage() do not insert new records in auxiliary tables if they are already there [v6r6p1] *Core: CHANGE: PromptUser - changed log level of the printout to NOTICE NEW: Base Client constructor arguments are passed to the RPCClient constructor *DMS: NEW: FTSRequest - added a prestage mechanism for source files NEW: FileCatalogClientCLI - added -f switch to the size command to use raw faile tables instead of storage usage tables NEW: FileCatalog - added orphan directory repair tool NEW: FIleCatalog - more counters to control the catalog sanity *WMS: FIX: SandboxStoreClient - no more kwargs tricks FIX: SandboxStoreClient returns sandbox file name in case of upload failure to allow failover FIX: dirac-pilot - fixed VO_%s_SW_DIR env variable in case of OSG *TS: FIX: TransformationManagerHandler - avoid multiple Operations() instantiation in getTransformationSummaryWeb() [v6r6] *Core CHANGE: getDNForUsername helper migrated from Core.Security.CS to Registry helper NEW: SiteSEMapping - new utilities getSitesGroupedByTierLevel(), getTier1WithAttachedTier2(), getTier1WithTier2 CHANGE: The DIRAC.Core.Security.CS is replaced by the Registry helper BUGFIX: dirac-install - properly parse += in .cfg files FIX: Graphs.Utilities - allow two lines input in makeDataFromCVS() FIX: Graphs - allow Graphs package usage if even matplotlib is not installed NEW: dirac-compile-externals will retrieve the Externals compilation scripts from it's new location in github (DIRACGrid/Externals) NEW: Possibility to define a thread-global credentials for DISET connections (for web framework) NEW: Logger - color output ( configurable ) NEW: dirac-admin-sort-cs-sites - to sort sites in the CS CHANGE: MessageClient(Factor) - added msgClient attribute to messages NEW: Core.Security.Properties - added JOB_MONITOR and USER_MANAGER properties *Configuration NEW: Registry - added getAllGroups() method *Framework NEW: SystemAdministratorClientCLI - possibility to define roothPath and lcgVersion when updating software *Accounting NEW: JobPlotter - added Normalized CPU plots to Job accounting FIX: DBUtils - plots going to greater granularity *DMS NEW: FileCatalog - storage usage info stored in all the directories, not only those with files NEW: FileCatalog - added utility to rebuild storage usage info from scratch FIX: FileCatalog - addMetadataField() allow generic types, e.g. string FIX: FileCatalog - path argument is normalized before usage in multiple methods FIX: FileCatalog - new metadata for files(directories) should not be there before for directories(files) NEW: FileCatalog - added method for rebuilding DirectoryUsage data from scratch NEW: FileCatalog - Use DirectoryUsage mechanism for both logical and physical storage CHANGE: FileCatalog - forbid removing non-empty directories BUGFIX: FileCatalogClientCLI - in do_ls() check properly the path existence FIX: FileCatalogClientCLI - protection against non-existing getCatalogCounters method in the LFC client FIX: DMS Agents - properly call superclass constructor with loadName argument FIX: ReplicaManager - in removeFile() non-existent file is marked as failed FIX: Make several classes pylint compliant: DataIntegrityHandler, DataLoggingHandler, FileCatalogHandler, StorageElementHandler, StorageElementProxyHandler, TransferDBMonitoringHandler FIX: LogUploadAgent - remove the OSError exception in __replicate() FIX: FileCatalogClientCLI - multiple check of proper command inputs, automatic completion of several commands with subcommands, automatic completion of file names CHANGE: FileCatalogClientCLI - reformat the output of size command FIX: dirac-admin-ban-se - allow to go over all options read/write/check for each SE NEW: StrategyHandler - new implementation to speed up file scheduling + better error reporting NEW: LcgFileCatalogProxy - moved from from LHCbDirac to DIRAC FIX: ReplicaManager - removed usage of obsolete "/Resources/StorageElements/BannedTarget" CHANGE: removed StorageUsageClient.py CHANGE: removed obsoleted ProcessingDBAgent.py *WMS CHANGE: RunNumber job parameter was removed from all the relevant places ( JDL, JobDB, etc ) NEW: dirac-pilot - add environment setting for SSH and BOINC CEs NEW: WMSAdministrator - get output for non-grid CEs if not yet in the DB NEW: JobAgent - job publishes BOINC parameters if any CHANGE: Get rid of LHCbPlatform everywhere except TaskQueueDB FIX: SiteDirector - provide list of sites to the Matcher in the initial query FIX: SiteDirector - present a list of all groups of a community to match TQs CHANGE: dirac-boinc-pilot dropped CHANGE: TaskQueueDirector does not depend on /LocalSite section any more CHANGE: reduced default delays for JobCleaningAgent CHANGE: limit the number of jobs received by JobCleaningAgent CHANGE: JobDB - use insertFields instead of _insert CHANGE: Matcher, TaskQueueDB - switch to use Platform rather than LHCbPlatform retaining LHCbPlatform compatibility BUGFIX: Matcher - proper reporting pilot site and CE CHANGE: JobManager - improved job Killing/Deleting logic CHANGE: dirac-pilot - treat the OSG case when jobs on the same WN all run in the same directory NEW: JobWrapper - added more status reports on different failures FIX: PilotStatusAgent - use getPilotProxyFromDIRACGroup() instead of getPilotProxyFromVOMSGroup() CHANGE: JobMonitoringHandler - add cutDate and condDict parameters to getJobGroup() NEW: JobMonitoringHandler - check access rights with JobPolicy when accessing job info from the web NEW: JobManager,JobWrapper - report to accounting jobs in Rescheduled final state if rescheduling is successful FIX: WMSAdministrator, SiteDirector - store only non-empty pilot output to the PilotDB NEW: added killPilot() to the WMSAdministrator interface, DiracAdmin and dirac-admin-kill-pilot command NEW: TimeLeft - renormalize time left using DIRAC Normalization if available FIX: JobManager - reconnect to the OptimizationMind in background if not yet connected CHANGE: JobManifest - use Operations helper NEW: JobCleaningAgent - delete logging records from JobLoggingDB when deleting jobs *RMS FIX: RequestDBFile - better exception handling in case no JobID supplied FIX: RequestManagerHandler - make it pylint compliant NEW: RequestProxyHandler - is forwarding requests from voboxes to central RequestManager. If central RequestManager is down, requests are dumped into file cache and a separate thread running in background is trying to push them into the central. CHANGE: Major revision of the code CHANGE: RequestDB - added index on SubRequestID in the Files table CHANGE: RequestClient - readRequestForJobs updated to the new RequetsClient structure *RSS NEW: CS.py - Space Tokens were hardcoded, now are obtained after scanning the StorageElements. *Resources FIX: SSHComputingElement - enabled multiple hosts in one queue, more debugging CHANGE: SSHXXX Computing Elements - define SSH class once in the SSHComputingElement NEW: SSHComputingElement - added option to define private key location CHANGE: Get rid of legacy methods in ComputingElement NEW: enable definition of ChecksumType per SE NEW: SSHBatch, SSHCondor Computing Elements NEW: SSHxxx Computing Elements - using remote control scripts to better capture remote command errors CHANGE: put common functionality into SSHComputingElement base class for all SSHxxx CEs NEW: added killJob() method tp all the CEs NEW: FileCatalog - take the catalog information info from /Operations CS section, if defined there, to allow specifications per VO *Interfaces CHANGE: Removed Script.initialize() from the API initialization CHANGE: Some general API polishing FIX: Dirac.py - when running in mode="local" any directory in the ISB would not get untarred, contrary to what is done in the JobWrapper *TS BUGFIX: TaskManager - bug fixed in treating tasks with input data FIX: TransformationCleaningAgent - properly call superclass constructor with loadName argument NEW: TransformationCleaningAgent - added _addExtraDirectories() method to extend the list of directories to clean in a subclass if needed CHANGE: TransformationCleaningAgent - removed usage of StorageUsageClient NEW: TransformationAgent is multithreaded now ( implementation moved from LHCbDIRAC ) NEW: added unit tests NEW: InputDataAgent - possibility to refresh only data registered in the last predefined period of time NEW: TransformationAgent(Client) - management of derived transformations and more ported from LHCbDIRAC BUGFIX: TransformationDB - wrong SQL statement generation in setFileStatusForTransformation() [v6r5p14] *Core NEW: Utilities - added Backports utility *WMS FIX: Use /Operations/JobScheduling section consistently, drop /Operations/Matching section NEW: Allow VO specific share correction plugins from extensions FIX: Executors - several fixes [v6r5p13] *WMS FIX: Executors - VOPlugin will properly send and receive the params NEW: Correctors can be defined in an extension FIX: Correctors - Properly retrieve info from the CS using the ops helper [v6r5p12] FIX: merged in patch v6r4p34 [v6r5p11] FIX: merged in patch v6r4p33 *Core FIX: MySQL - added offset argument to buildConditions() [v6r5p10] FIX: merged in patch v6r4p32 [v6r5p9] FIX: merged in patch v6r4p30 [v6r5p8] FIX: merged in patch v6r4p29 [v6r5p7] FIX: merged in patch v6r4p28 [v6r5p6] FIX: merged in patch v6r4p27 *Transformation BUGFIX: TransformationDB - StringType must be imported before it can be used *RSS NEW: CS.py - Space Tokens were hardcoded, now are obtained after scanning the StorageElements. [v6r5p5] FIX: merged in patch v6r4p26 [v6r5p4] FIX: merged in patch v6r4p25 [v6r5p3] *Transformation FIX: merged in patch v6r4p24 [v6r5p2] *Web NEW: includes DIRACWeb tag web2012092101 [v6r5p1] *Core BUGFIX: ExecutorMindHandler - return S_OK() in the initializeHandler FIX: OptimizationMindHandler - if the manifest is not dirty it will not be updated by the Mind *Configuration NEW: Resources helper - added getCompatiblePlatform(), getDIRACPlatform() methods *Resources FIX: SSHComputingElement - add -q option to ssh command to avoid banners in the output FIX: BOINCComputingElement - removed debugging printout FIX: ComputingElement - use Platform CS option which will be converted to LHCbPlatform for legacy compatibility *DMS FIX: RequestAgentBase - lowering loglevel from ALWAYS to INFO to avoid flooding SystemLogging *WMS: FIX: SiteDirector - provide CE platform parameter when interrogating the TQ FIX: GridPilotDirector - publish pilot OwnerGroup rather than VOMS role FIX: WMSUtilities - add new error string into the parsing of the job output retrieval [v6r5] NEW: Executor framework *Core NEW: MySQL.py - added Test case for Time.dateTime time stamps NEW: MySQL.py - insertFields and updateFields can get values via Lists or Dicts NEW: DataIntegrityDB - use the new methods from MySQL and add test cases NEW: DataIntegrityHandler - check connection to DB and create tables (or update their schema) NEW: DataLoggingDB - use the new methods from MySQL and add test cases NEW: DataLoggingHandler - check connection to DB and create tables (or update their schema) FIX: ProcessPool - killing stuck workers after timeout CHANGE: DB will throw a RuntimeException instead of a sys.exit in case it can't contact the DB CHANGE: Several improvements on DISET CHANGE: Fixed all DOS endings to UNIX CHANGE: Agents, Services and Executors know how to react to CSSection/Module and react accordingly NEW: install tools are updated to deal with executors FIX: dirac-install - add -T/--Timeout option to define timeout for distribution downloads NEW: dirac-install - added possibility of defining dirac-install's global defaults by command line switch BUGFIX: avoid PathFinder.getServiceURL and use Client class ( DataLoggingClient,LfcFileCatalogProxyClient ) FIX: MySQL - added TIMESTAMPADD and TIMESTAMPDIFF to special values not to be scaped by MySQL NEW: ObjectLoader utility CHANGE: dirac-distribution - added global defaults flag and changed the flag to -M or --defaultsURL FIX: Convert to string before trying to escape value in MySQL NEW: DISET Services - added PacketTimeout option NEW: SystemLoggingDB - updated to use the renewed MySQL interface and SQL schema NEW: Added support for multiple entries in /Registry/DefaultGroup, for multi-VO installations CHANGE: Component installation procedure updated to cope with components inheriting Modules CHANGE: InstallTools - use dirac- command in runit run scripts FIX: X509Chain - avoid a return of error when the group is not valid FIX: MySQL - reduce verbosity of log messages when high level methods are used CHANGE: Several DB classes have been updated to use the MySQL buildCondition method NEW: MySQL - provide support for greater and smaller arguments to all MySQL high level methods FIX: Service.py - check all return values from all initializers *Configuration CHANGE: By default return option and section lists ordered as in the CS NEW: ConfigurationClient - added function to refresh remote configuration *Framework FIX: Registry.findDefaultGroup will never return False CHANGE: ProxyManager does not accept proxies without explicit group CHANGE: SystemAdministratorHandler - force refreshing the configuration after new component setup *RSS CHANGE: removed code execution from __init__ CHANGE: removed unused methods NEW: Log all policy results *Resources NEW: updated SSHComputingElement which allows multiple job submission FIX: SGETimeLeft - better parsing of the batch system commands output FIX: InProcessComputingElement - when starting a new job discard renewal of the previous proxy NEW: BOINCComputingElement - new CE client to work with the BOINC desktop grid infrastructure *WMS CHANGE: WMS Optimizers are now executors CHANGE: SandboxStoreClient can directly access the DB if available CHANGE: Moved JobDescription and improved into JobManifest FIX: typo in JobLoggingDB NEW: JobState/CachedJobState allow access to the Job via DB/JobStateSync Service automatically BUGFIX: DownloadInputData - when not enough disk space, message was using "buffer" while it should be using "data" FIX: the sandboxmetadataDB explosion when using the sandboxclient without direct access to the DB NEW: Added support for reset/reschedule in the OptimizationMind CHANGE: Whenever a DB is not properly initialized it will raise a catchable RuntimeError exception instead of silently returning FIX: InputDataResolution - just quick mod for easier extensibility, plus removed some LHCb specific stuff NEW: allow jobids in a file in dirac-wms-job-get-output NEW: JobManager - zfill in %n parameter substitution to allow alphabetical sorting NEW: Directors - added checking of the TaskQueue limits when getting eligible queues CHANGE: Natcher - refactor to simpify the logic, introduced Limiter class CHANGE: Treat MaxCPUTime and CPUTime the same way in the JDL to avoid confusion NEW: SiteDirector - added options PilotScript, MaxPilotsToSubmit, MaxJobsInFillMode BUGFIX: StalledJobAgent - use cpuNormalization as float, not string FIX: Don't kill an executor if a task has been taken out from it NEW: dirac-boinc-pilot - pilot script to be used on the BOINC volunteer nodes FIX: SiteDirector - better handling of tokens and filling mode NEW: Generic pilot identities are automatically selected by the TQD and the SiteDirector if not explicitly defined in /Pilot/GenericDN and GenericGroup NEW: Generic pilot groups can have a VO that will be taken into account when selecting generic credentials to submit pilots NEW: Generic pilots that belong to a VO can only match jobs from that VO NEW: StalledJobAgent - added rescheduling of jobs stuck in Matched or Rescheduled status BUGFIX: StalledJobAgent - default startTime and endTime to "now", avoid None value NEW: JobAgent - stop after N failed matching attempts (nothing to do), use StopAfterFailedMatches option CHANGE: JobAgent - provide resource description as a dictionary to avoid extra JDL parsing by the Matcher CHANGE: Matcher - report pilot info once instead of sending it several times from the job CHANGE: Matcher - set the job site instead of making a separate call to JobStateUpdate NEW: Matcher - added Matches done and matches OK statistics NEW: TaskQueue - don't delete fresh task queues. Wait 5 minutes to do so. CHANGE: Disabled TQs can also be matched, if no jobs are there, a retry will be triggered *Transformation FIX: TransformationAgent - a small improvement: now can pick the prods status to handle from the CS, plus few minor corrections (e.g. logger messages) FIX: TransformationCLI - take into accout possible failures in resetFile command *Accounting NEW: AccountingDB - added retrieving RAW records for internal stuff FIX: AccountingDB - fixed some logic for readonly cases CHANGE: Added new simpler and faster bucket insertion mechanism NEW: Added more info when rebucketing FIX: Calculate the rebucket ETA using remaining records to be processed instead of the total records to be processed FIX: Plots with no data still carry the plot name *DMS NEW: SRM2Storage - added retry in the gfal calls NEW: added new FTSCleaningAgent cleaning up TransferDB tables FIX: DataLoggingClient and DataLoggingDB - tests moved to separate files CHANGE: request agents cleanup *RMS CHANGE: Stop using RequestAgentMixIn in the request agents [v6r4p34] *DMS BUGFIX: FileCatalogCLI - fixed wrong indentation CHANGE: RegistrationTask - removed some LHCb specific defaults [v6r4p33] *DMS CHANGE: FTSRequest - be more verbose if something is wrong with file [v6r4p32] *WMS FIX: StalledJobAgent - avoid exceptions in the stalled job accounting reporting *DMS NEW: FTSMonitorAgent - handling of expired FTS jobs *Interfaces CHANGE: Dirac.py - attempt to retrieve output sandbox also for Completed jobs in retrieveRepositorySandboxes() [v6r4p30] *Core BUGFIX: dirac-admin-bdii-ce-voview - proper check of the result structure *Interfaces FIX: Dirac.py, Job.py - allow to pass environment variables with special characters *DMS NEW: FileCatalogCLI - possibility to sort output in the ls command *WMS: FIX: JobWrapper - interpret environment variables with special characters [v6r4p29] *RMS BUGFIX: RequestDBMySQL - wrong indentation in __updateSubRequestFiles() [v6r4p28] *Interfaces CHANGE: Dirac.py, DiracAdmin.py - remove explicit timeout on RPC client instantiation *RSS FIX: CS.py - fix for updated CS location (backward compatible) *DMS BUGFIX: StrategyHandler - bug fixed determineReplicationTree() FIX: FTSRequest - add checksum string to SURLs file before submitting an FTS job *WMS FIX: JobWrapper - protection for double quotes in JobName CHANGE: SiteDirector - switched some logging messages from verbose to info level *RMS NEW: Request(Client,DBMySQL,Manager) - added readRequestsForJobs() method [v6r4p27] *DMS FIX: SRM2Storage - removed hack for EOS (fixed server-side) *Transformation CHANGE: TransformationClient - limit to 100 the number of transformations in getTransformations() NEW: TransformationAgent - define the transformations type to use in the configuration *Interfaces FIX: Job.py - fix for empty environmentDict (setExecutionEnv) [v6r4p26] *Transformation BUGFIX: TransformationClient - fixed calling sequence in rpcClient.getTransformationTasks() NEW: TransformationClient - added log messages in verbose level. [v6r4p25] *DMS BUGFIX: StrategyHandler - sanity check for wrong replication tree [v6r4p24] *Core NEW: MySQL - add 'offset' argument to the buildCondition() *Transformation FIX: TransformationAgent - randomize the LFNs for removal/replication case when large number of those CHANGE: TransformationClient(DB,Manager) - get transformation files in smaller chunks to improve performance FIX: TransformationAgent(DB) - do not return redundant LFNs in getTransformationFiles() [v6r4p23] *Web NEW: includes DIRACWeb tag web2012092101 [v6r4p22] *DMS FIX: SRM2Storage - fix the problem with the CERN-EOS storage [v6r4p21] *Core BUGFIX: SGETimeLeft - take into account dd:hh:mm:ss format of the cpu consumed [v6r4p20] *WMS BUGFIX: PilotDirector, GridPilotDirector - make sure that at least 1 pilot is to be submitted BUGFIX: GridPilotDirector - bug on how pilots are counted when there is an error in the submit loop. BUGFIX: dirac-pilot - proper install script installation on OSG sites [v6r4p19] *RMS FIX: RequestDBMySQL - optimized request selection query [v6r4p18] *Configuration BUGFIX: CE2CSAgent.py - the default value must be set outside the loop *DMS NEW: dirac-dms-create-replication-request BUGFIX: dirac-dms-fts-submit, dirac-dms-fts-monitor - print out error messages *Resources BUGFIX: TorqueComputingElement.py, plus add UserName for shared Queues *WMS BUGFIX: JobManagerHandler - default value for pStart (to avoid Exception) [v6r4p17] *Core FIX: dirac-configure - setup was not updated in dirac.cfg even with -F option FIX: RequestHandler - added fix for Missing ConnectionError *DMS FIX: dirac-dms-clean-directory - command fails with `KeyError: 'Replicas'`. *WMS FIX: SiteDirector - adapt to the new method in the Matcher getMatchingTaskQueue FIX: SiteDirector - added all SubmitPools to TQ requests [v6r4p16] *Core: FIX: dirac-install - bashrc/cshrc were wrongly created when using versionsDir *Accounting CHANGE: Added new simpler and faster bucket insertion mechanism NEW: Added more info when rebucketing *WMS CHANGE: Matcher - refactored to take into account job limits when providing info to directors NEW: JoAgent - reports SubmitPool parameter if applicable FIX: Matcher - bad codition if invalid result [v6r4p15] *WMS FIX: gLitePilotDirector - fix the name of the MyProxy server to avoid crasehs of the gLite WMS *Transformation FIX: TaskManager - when the file is on many SEs, wrong results were generated [v6r4p13] *DMS FIX: dirac-admin-allow-se - added missing interpreter line [v6r4p12] *DMS CHANGE: RemovalTask - for DataManager shifter change creds after failure of removal with her/his proxy. *RSS NEW: Added RssConfiguration class FIX: ResourceManagementClient - Fixed wrong method name [v6r4p11] *Core FIX: GGUSTicketsClient - GGUS SOAP URL updated *DMS BUGFIX: ReplicaManager - wrong for loop *RequestManagement BUGFIX: RequestClient - bug fix in finalizeRequest() *Transformation FIX: TaskManager - fix for correctly setting the sites (as list) [v6r4p10] *RequestManagement BUGFIX: RequestContainer - in addSubrequest() function *Resources BUGFIX: SRM2Storage - in checksum type evaluation *ResourceStatusSystem BUGFIX: InfoGetter - wrong import statement *WMS BUGFIX: SandboxMetadataDB - __init__() can not return a value [v6r4p9] *DMS CHANGE: FailoverTransfer - ensure the correct execution order of the subrequests [v6r4p8] Bring in fixes from v6r3p17 *Core: FIX: Don't have the __init__ return True for all DBs NEW: Added more protection for exceptions thrown in callbacks for the ProcessPool FIX: Operations will now look in 'Defaults' instead of 'Default' *DataManagement: FIX: Put more protection in StrategyHandler for neither channels not throughput read out of TransferDB FIX: No JobIDs supplied in getRequestForJobs function for RequestDBMySQL taken into account FIX: Fix on getRequestStatus CHANGE: RequestClient proper use of getRequestStatus in finalizeRequest CHANGE: Refactored RequestDBFile [v6r4p7] *WorkloadManagement FIX: SandboxMetadataDB won't explode DIRAC when there's no access to the DB CHANGE: Whenever a DB fails to initialize it raises a catchable exception instead of just returning silently *DataManagement CHANGE: Added Lost and Unavailable to the file metadata [v6r4p6] Bring fixes from v6r4p6 [v6r4p5] *Configuration NEW: Added function to generate Operations CS paths *Core FIX: Added proper ProcessPool checks and finalisation *DataManagement FIX: don't set Files.Status to Failed for non-existign files, failover transfers won't go FIX: remove classmethods here and there to unblock requestHolder CHANGE: RAB, TA: change task timeout: 180 and 600 (was 600 and 900 respectively) FIX: sorting replication tree by Ancestor, not hopAncestorgit add DataManagementSystem/Agent/TransferAgent.py NEW: TA: add finalize CHANGE: TransferAgent: add AcceptableFailedFiles to StrategyHandler to ban FTS channel from scheduling FIX: if there is no failed files, put an empty dict *RSS FIX: RSS is setting Allowed but the StorageElement checks for Active *Workflows FIX: Part of WorfklowTask rewritten to fix some issues and allow 'ANY' as site *Transformation FIX: Wrong calls to TCA::cleanMetadataCatalogFiles [v6r4p4] *Core FIX: Platform.py - check if Popen.terminate is available (only from 2.6) [v6r4p3] *Core FIX: ProcessPool with watchdog and timeouts - applied in v6r3 first [v6r4p2] *StorageManagement BUGFIX: StorageElement - staging is a Read operation and should be allowed as such *WMS BUGFIX: InProcessComputingElement, JobAgent - proper return status code from the job wrapper *Core FIX: Platform - manage properly the case of exception in the ldconfig execution [v6r4p1] *DMS FIX: TransferDB.getChannelObservedThroughput - the channelDict was created in a wrong way *RSS FIX: ResourceStatus was not returning Allowed by default [v6r4] *Core FIX: dirac-install-db.py: addDatabaseOptionsToCS has added a new keyed argument NEW: SGETimeLeft.py: Support for SGE backend FIX: If several extensions are installed, merge ConfigTemplate.cfg NEW: Service framework - added monitoring of file descriptors open NEW: Service framework - Reduced handshake timeout to prevent stuck threads NEW: MySQL class with new high level methods - buildCondition,insertFields,updateFields deleteEntries, getFields, getCounters, getDistinctAttributeValues FIX: ProcessPool - fixes in the locking mechanism with LockRing, stopping workers when the parent process is finished FIX: Added more locks to the LockRing NEW: The installation tools are updated to install components by name with the components module specified as an option *DMS FIX: TransferDB.py - speed up the Throughput determination NEW: dirac-dms-add-files: script similar to dirac-dms-remove-files, allows for 1 file specification on the command line, using the usual dirac-dms-add-file options, but also can take a text file in input to upload a bunch of files. Exit code is 0 only if all was fine and is different for every error found. NEW: StorageElementProxy- support for data downloading with http protocol from arbitrary storage, needed for the web data download BUGFIX: FileCatalogCLI - replicate operation does a proper replica registration ( closes #5 ) FIX: ReplicaManager - __cleanDirectory now working and thus dirac-dms-clean-directory *WMS NEW: CPU normalization script to run a quick test in the pilot, used by the JobWrapper to report the CPU consumption to the accounting FIX: StalledJobAgent - StalledTimeHours and FailedTimeHours are read each cycle, refer to the Watchdog heartBeat period (should be renamed); add NormCPUTime to Accounting record NEW: SiteDirector - support for the operation per VO in multi-VO installations FIX: StalledJobAgent - get ProcessingType from JDL if defined BUGFIX: dirac-wms-job-peek - missing printout in the command NEW: SiteDirector - take into account the number of already waiting pilots when evaluating the number of pilots to submit FIX: properly report CPU usage when the Watchdog kill the payload. *RSS BUGFIX: Result in ClientCache table is a varchar, but the method was getting a datetime NEW: CacheFeederAgent - VOBOX and SpaceTokenOccupancy commands added (ported from LHCbDIRAC) CHANGE: RSS components get operational parameters from the Operations handler *DataManagement FIX: if there is no failed files, put an empty dict *Transformation FIX: Wrong calls to TCA::cleanMetadataCatalogFiles [v6r3p19] *WMS FIX: gLitePilotDirector - fix the name of the MyProxy server to avoid crashes of the gLite WMS [v6r3p18] *Resources BUGFIX: SRM2Storage - in checksum type evaluation [v6r3p17] *DataManagement FIX: Fixes issues #783 and #781. Bugs in ReplicaManager removePhisicalReplica and getFilesFromDirectory FIX: Return S_ERROR if missing jobid arguments NEW: Checksum can be verified during FTS and SRM2Storage [v6r3p16] *DataManagement FIX: better monitoring of FTS channels FIX: Handle properly None value for channels and bandwidths *Core FIX: Properly calculate the release notes if there are newer releases in the release.notes file [v6r3p15] *DataManagement FIX: if there is no failed files, put an empty dict *Transformation FIX: Wrong calls to TCA::cleanMetadataCatalogFiles [v6r3p14] * Core BUGFIX: ProcessPool.py: clean processing and finalisation BUGFIX: Pfn.py: don't check for 'FileName' in pfnDict * DMS NEW: dirac-dms-show-fts-status.py: script showing last hour history for FTS channels NEW: TransferDBMonitoringHandler.py: new function exporting FST channel queues BUGFIX: TransferAgent.py,RemovalAgent.py,RegistrationAgent.py - unlinking of temp proxy files, corection of values sent to gMonitor BUGFIX: StrategyHandler - new config option 'AcceptableFailedFiles' to unblock scheduling for channels if problematic transfers occured for few files NEW: TransferAgent,RemovalAgent,RegistrationAgent - new confing options for setting timeouts for tasks and ProcessPool finalisation BUGFIX: ReplicaManager.py - reverse sort of LFNs when deleting files and directories to avoid blocks NEW: moved StrategyHandler class def to separate file under DMS/private * TMS FIX: TransformationCleaningAgent.py: some refactoring, new way of disabling/enabline execution by 'EnableFlag' config option [v6r3p13] *Core FIX: Added proper ProcessPool checks and finalisation *DataManagement FIX: don't set Files.Status to Failed for non-existign files, failover transfers won't go FIX: remove classmethods here and there to unblock requestHolder CHANGE: RAB, TA: change task timeout: 180 and 600 (was 600 and 900 respectively) FIX: sorting replication tree by Ancestor, not hopAncestorgit add DataManagementSystem/Agent/TransferAgent.py NEW: TA: add finalize CHANGE: TransferAgent: add AcceptableFailedFiles to StrategyHandler to ban FTS channel from scheduling [v6r3p12] *Core FIX: Platform.py - check if Popen.terminate is available (only from 2.6) [v6r3p11] *Core FIX: ProcessPool with watchdog and timeouts [v6r3p10] *StorageManagement BUGFIX: StorageElement - staging is a Read operation and should be allowed as such *WMS BUGFIX: InProcessComputingElement, JobAgent - proper return status code from the job wrapper *Core FIX: Platform - manage properly the case of exception in the ldconfig execution [v6r3p9] *DMS FIX: TransferDB.getChannelObservedThroughput - the channelDict was created in a wrong way [v6r3p8] *Web CHANGE: return back to the release web2012041601 [v6r3p7] *Transformation FIX: TransformationCleaningAgent - protection from deleting requests with jobID 0 [v6r3p6] *Core FIX: dirac-install-db - proper key argument (follow change in InstallTools) FIX: ProcessPool - release all locks every time WorkignProcess.run is executed, more fixes to come FIX: dirac-configure - for Multi-Community installations, all vomsdir/vomses files are now created *WMS NEW: SiteDirector - add pilot option with CE name to allow matching of SAM jobs. BUGFIX: dirac-pilot - SGE batch ID was overwriting the CREAM ID FIX: PilotDirector - protect the CS master if there are at least 3 slaves NEW: Watchdog - set LocalJobID in the SGE case [v6r3p5] *Core: BUGFIX: ProcessPool - bug making TaskAgents hang after max cycles BUGFIX: Graphs - proper handling plots with data containing empty string labels FIX: GateWay - transfers were using an old API FIX: GateWay - properly calculate the gateway URL BUGFIX: Utilities/Pfn.py - bug in pfnunparse() when concatenating Path and FileName *Accounting NEW: ReportGenerator - make AccountingDB readonly FIX: DataCache - set daemon the datacache thread BUGFIX: BasePlotter - proper handling of the Petabyte scale data *DMS: BUGFIX: TransferAgent, RegistrationTask - typos [v6r3p4] *DMS: BUGFIX: TransferAgent - wrong value for failback in TA:execute [v6r3p3] *Configuration BUGFIX: Operations helper - typo *DMS: FIX: TransferAgent - change the way of redirecting request to task [v6r3p2] *DMS FIX: FTSRequest - updating metadata for accouting when finalizing FTS requests *Core FIX: DIRAC/__init__.py - default version is set to v6r3 [v6r3p1] *WMS CHANGE: Use ResourcesStatus and Resources helpers in the InputDataAgent logic *Configuration NEW: added getStorageElementOptions in Resources helper *DMS FIX: resourceStatus object created in TransferAgent instead of StrategyHandler [v6r3] *Core NEW: Added protections due to the process pool usage in the locking logic *Resources FIX: LcgFileCatalogClient - reduce the number of retries: LFC_CONRETRY = 5 to avoid combined catalog to be stuck on a faulty LFC server *RSS BUGFIX: ResourceStatus - reworked helper to keep DB connections *DMS BUGFIX: ReplicaManager::CatalogBase::_callFileCatalogFcnSingleFile() - wrong argument *RequestManagement FIX: TaskAgents - set timeOut for task to 10 min (15 min) NEW: TaskAgents - fill in Error fields in case of failing operations *Interfaces BUGFIX: dirac-wms-select-jobs - wrong use of the Dirac API [v6r2p9] *Core FIX: dirac-configure - make use of getSEsForSite() method to determine LocalSEs *WMS NEW: DownloadInputData,InputDataByProtocol - check Files on Tape SEs are on Disk cache before Download or getturl calls from Wrapper CHANGE: Matcher - add Stalled to "Running" Jobs when JobLimits are applied CHANGE: JobDB - allow to specify required platform as Platform JDL parameter, the specified platform is taken into account even without /Resources/Computing/OSCompatibility section *DMS CHANGE: dirac-admin-allow(ban)-se - removed lhcb-grid email account by default, and added switch to avoid sending email FIX: TaskAgents - fix for non-existing files FIX: change verbosity in failoverReplication FIX: FileCatalog - remove properly metadata indices BUGFIX: FileManagerBase - bugfix in the descendants evaluation logic FIX: TransferAgent and TransferTask - update Files.Status to Failed when ReplicaManager.replicateAndRegister will fail completely; when no replica is available at all. *Core FIX: dirac-pilot - default lcg bindings version set to 2012-02-20 [v6r2p8] *DMS: CHANGE: TransferAgent - fallback to task execution if replication tree is not found [v6r2p7] *WMS BUGFIX: SiteDirector - wrong CS option use: BundleProxy -> HttpProxy FIX: SiteDirector - use short lines in compressed/encoded files in the executable python script [v6r2p6] *DataManagement FIX: Bad logic in StrategyHandler:MinimiseTotalWait *Core CHANGE: updated GGUS web portal URL *RSS BUGFIX: meta key cannot be reused, it is popped from dictionary *Framework FIX: The Gateway service does not have a handler NEW: ConfingTemplate entry for Gateway FIX: distribution notes allow for word wrap *WorkloadManagement FIX: avoid unnecessary call if no LFN is left in one of the SEs FIX: When Uploading job outputs, try first Local SEs, if any [v6r2p5] *RSS BUGFIX: several minor bug fixes *RequestManagement BUGFIX: RequestDBMySQL - removed unnecessary request type check *DMS BUGFIX: FileCatalogClienctCLI - wrong evaluation of the operation in the find command NEW: FileCatalog - added possibility to remove specified metadata for a given path BUGFIX: ReplicaManager - wrong operation order causing failure of UploadLogFile module *Core NEW: dirac-install - generate cshrc DIRAC environment setting file for the (t)csh *Interfaces CHANGE: Job - added InputData to each element in the ParametricInputData *WMS CHANGE: dirac-jobexec - pass ParametericInputData to the workflow as a semicolon separated string [v6r2p4] *WMS BUGFIX: StalledJobAgent - protection against jobs with no PilotReference in their parameters BUGFIX: WMSAdministratorHandler - wrong argument type specification for getPilotInfo method *StorageManagement BUGFIX: RequestFinalizationAgent - no method existence check when calling RPC method [v6r2p3] *WMS CHANGE: Matcher - fixed the credentials check in requestJob() to simplify it *ConfigurationSystem CHANGE: Operations helper - fix that allow no VO to be defined for components that do not need it *Core BUGFIX: InstallTools - when applying runsvctrl to a list of components make sure that the config server is treated first and the sysadmin service - last [v6r2p2] *WMS BUGFIX: Matcher - restored logic for checking private pilot asking for a given DN for belonging to the same group with JOB_SHARING property. [v6r2p1] *RequestManagementSystem BUGFIX: RequestCleaningAgent - missing import of the "second" interval definition [v6r2] *General FIX: replaced use of exec() python statement in favor of object method execution *Accounting CHANGE: Accounting 'byte' units are in powers of 1000 instead of powers of 1024 (closes #457) *Core CHANGE: Pfn.py - pfnparse function rewritten for speed up and mem usage, unit test case added FIX: DISET Clients are now thread-safe. Same clients used twice in different threads was not closing the previous connection NEW: reduce wait times in DISET protocol machinery to improve performance NEW: dirac-fix-mysql-script command to fix the mysql start-up script for the given installation FIX: TransferClient closes connections properly FIX: DISET Clients are now thread-safe. Same client used twice in different threads will not close the previous connection CHANGE: Beautification and reduce wait times to improve performance NEW: ProcessPool - added functionality to kill all children processes properly when destroying ProcessPool objects NEW: CS Helper for LocalSite section, with gridEnv method NEW: Grid module will use Local.gridEnv if nothing passed in the arguments CHANGE: Add deprecated sections in the CS Operations helper to ease the transition FIX: dirac-install - execute dirac-fix-mysql-script, if available, to fix the mysql.server startup script FIX: dirac-distribution - Changed obsoleted tar.list file URL FIX: typo in dirac-admin-add-host in case of error CHANGE: dirac-admin-allow(ban)-se - use diracAdmin.sendMail() instead of NotificationClient.sendMail() *Framework BUGFIX: UserProfileDB - no more use of "type" variable as it is a reserved keyword *RequestManagement: FIX: RequestDBFile - more consistent treatment of requestDB Path FIX: RequestMySQL - Execution order is evaluated based on not Done state of subrequests NEW: RequestCleaningAgent - resetting Assigned requests to Waiting after a configurable period of time *RSS CHANGE: RSS Action now inherits from a base class, and Actions are more homogeneous, they all take a uniform set of arguments. The name of modules has been changed from PolType to Action as well. FIX: CacheFeederAgent - too verbose messages moved to debug instead of info level BUGFIX: fixed a bug preventing RSS clients to connect to the services FIX: Proper services synchronization FIX: Better handling of exceptions due to timeouts in GOCDBClient FIX: RSS.Notification emails are sent again FIX: Commands have been modified to return S_OK, S_ERROR inside the Result dict. This way, policies get a S_ERROR / S_OK object. CacheFeederAgent has been updated accordingly. FIX: allow clients, if db connection fails, to reconnect ( or at least try ) to the servers. CHANGE: access control using CS Authentication options. Default is SiteManager, and get methods are all. BUGFIX: MySQLMonkey - properly escaped all parameters of the SQL queries, other fixes. NEW: CleanerAgent renamed to CacheCleanerAgent NEW: Updated RSS scripts, to set element statuses and / or tokens. NEW: Added a new script, dirac-rss-synch BUGFIX: Minor bugfixes spotted on the Web development FIX: Removed useless decorator from RSS handlers CHANGE: ResourceStatus helper tool moved to RSS/Client directory, no RSS objects created if the system is InActive CHANGE: Removed ClientFastDec decorator, using a more verbose alternative. CHANGE: Removed useless usage of kwargs on helper functions. NEW: added getSESitesList method to RSSClient FIX: _checkFloat() checks INTEGERS, not datetimes *DataManagement CHANGE: refactoring of DMS agents executing requests, allow requests from arbitrary users NEW: DFC - allow to specify multiple replicas, owner, mode when adding files CHANGE: DFC - optimization of the directory size evaluation NEW: Added CREATE TEMPORARY TABLES privilege to FileCatalogDB CHANGE: DFC - getCatalogCounters() update to show numbers of directories NEW: lfc_dfc_copy script to migrate data from LFC to DFC FIX: dirac-dms-user-lfns - fixed the case when the baseDir is specified FIX: FTS testing scripts were using sys.argv and getting confused if options are passed NEW: DFC - use DirectoryUsage tables for the storage usage evaluations NEW: DFC - search by metadata can be limited to a given directory subtree NEW: DFC - search by both directory and file indexed metadata BUGFIX: DFC - avoid crash if no directories or files found in metadata query NEW: DFC FileCatalogHandler - define database location in the configuration NEW: DFC - new FileCatalogFactory class, possibility to use named DFC services FIX: FTSMonitor, FTSRequest - fixes in handling replica registration, setting registration requests in FileToCat table for later retry FIX: Failover registration request in the FTS agents. FIX: FTSMonitor - enabled to register new replicas if even the corresponding request were removed from the RequestManagement FIX: StorageElement - check if SE has been properly initialized before executing any method CHANGE: LFC client getReplica() - make use of the new bulk method lfc.lfc_getreplicasl() FIX: LFC client - protect against getting None in lfc.lfc_readdirxr( oDirectory, "" ) FIX: add extra protection in dump method of StorageElement base class CHANGE: FailoverTransfer - create subrequest per catalog if more than one catalog *Interface NEW: Job.py - added method to handle the parametric parameters in the workflow. They are made available to the workflow_commons via the key 'GenericParameters'. FIX: Dirac.py - fix some type checking things FIX: Dirac.py - the addFile() method can now register to more than 1 catalog. *WMS FIX: removed dependency of the JobSchedulingAgent on RSS. Move the getSiteTier functionality to a new CS Helper. FIX: WMSAdministratorHandler - Replace StringType by StringTypes in the export methods argument type FIX: JobAgent - Set explicitly UseServerCertificate to "no" for the job executable NEW: dirac-pilot - change directory to $OSG_WN_TMP on OSG sites FIX: SiteDirector passes jobExecDir to pilot, this defaults to "." for CREAM CEs. It can be set in the CS. It will not make use of $TMPDIR in this case. FIX: Set proper project and release version to the SiteDirector NEW: Added "JobDelay" option for the matching, refactored and added CS options to the matcher FIX: Added installation as an option to the pilots and random MyProxyServer NEW: Support for parametric jobs with parameters that can be of List type *Resources NEW: Added SSH Grid Engine Computing Element NEW: Added SSH Computing Element FIX: make sure lfc client will not try to connect for several days *Transformation FIX: TransformationDB - in setFileStatusForTransformation() reset ErrorCount to zero if "force" flag and the new status is "unused" NEW: TransformationDB - added support for dictionary in metadata for the InputDataQuery mechanism [v6r1p13] *WMS FIX: JobSchedulingAgent - backported from v6r2 use of Resources helper [v6r1p12] *Accounting FIX: Properly delete cached plots *Core FIX: dirac-install - run externals post install after generating the versions dir [v6r1p11] *Core NEW: dirac-install - caches locally the externals and the grid bundle FIX: dirac-distribution - properly generate releasehistory and releasenotes [v6r1p10] *WorloadManagement FIX: JobAgent - set UseServerCertificate option "no" for the job executable [v6r1p9] *Core FIX: dirac-configure - set the proper /DIRAC/Hostname when defining /LocalInstallation/Host *DataManagement FIX: dirac-dms-user-lfns - fixed the case when the baseDir is specified BUGFIX: dirac-dms-remove-files - fixed crash in case of returned error report in a form of dictionary [v6r1p8] *Web FIX: restored Run panel in the production monitor *Resources FIX: FileCatalog - do not check existence of the catalog client module file [v6r1p7] *Web BUGFIX: fixed scroll bar in the Monitoring plots view [v6r1p6] *Core FIX: TransferClient closes connections properly [v6r1p5] *Core FIX: DISET Clients are now thread-safe. Same clients used twice in different threads was not closing the previous connection NEW: reduce wait times in DISET protocol machinery to improve performance [v6r1p4] *RequestManagement BUGFIX: RequestContainer - in isSubRequestDone() treat special case for subrequests with files *Transformation BUGFIX: TransformationCleaningAgent - do not clear requests for tasks with no associated jobs [v6r1p3] *Framework NEW: Pass the monitor down to the request RequestHandler FIX: Define the service location for the monitor FIX: Close some connections that DISET was leaving open [v6r1p2] *WorkloadManagement BUGFIX: JobSchedulingAgent - use getSiteTiers() with returned direct value and not S_OK *Transformation BUGFIX: Uniform use of the TaskManager in the RequestTaskAgent and WorkflowTaskAgent [v6r1p1] *RSS BUGFIX: Alarm_PolType now really send mails instead of crashing silently. [v6r1] *RSS CHANGE: Major refactoring of the RSS system CHANGE: DB.ResourceStatusDB has been refactored, making it a simple wrapper round ResourceStatusDB.sql with only four methods by table ( insert, update, get & delete ) CHANGE: DB.ResourceStatusDB.sql has been modified to support different statuses per granularity. CHANGE: DB.ResourceManagementDB has been refactored, making it a simple wrapper round ResourceStatusDB.sql with only four methods by table ( insert, update, get & delete ) CHANGE: Service.ResourceStatusHandler has been refactored, removing all data processing, making it an intermediary between client and DB. CHANGE: Service.ResourceManagementHandler has been refactored, removing all data processing, making it an intermediary between client and DB. NEW: Utilities.ResourceStatusBooster makes use of the 'DB primitives' exposed on the client and does some useful data processing, exposing the new functions on the client. NEW: Utilities.ResourceManagementBooster makes use of the 'DB primitives' exposed on the client and does some useful data processing, exposing the new functions on the client. CHANGE: Client.ResourceStatusClient has been refactorerd. It connects automatically to DB or to the Service. Exposes DB and booster functions. CHANGE: Client.ResourceManagementClient has been refactorerd. It connects automatically to DB or to the Service. Exposes DB and booster functions. CHANGE: Agent.ClientsCacheFeederAgent renamed to CacheFeederAgent. The name was not accurate, as it also feeds Accouting Cache tables. CHANGE: Agent.InspectorAgent, makes use of automatic API initialization. CHANGE: Command. refactor and usage of automatic API initialization. CHANGE: PolicySystem.PEP has reusable client connections, which increase significantly performance. CHANGE: PolicySystem.PDP has reusable client connections, which increase significantly performance. NEW: Utilities.Decorators are syntactic sugar for DB, Handler and Clients. NEW: Utilities.MySQLMonkey is a mixture of laziness and refactoring, in order to generate the SQL statements automatically. Not anymore sqlStatemens hardcoded on the RSS. NEW: Utilities.Validator are common checks done through RSS modules CHANGE: Utilities.Synchronizer syncs users and DIRAC sites CHANGE: cosmetic changes everywhere, added HeadURL and RCSID CHANGE: Removed all the VOExtension logic on RSS BUGFIX: ResourceStatusHandler - getStorageElementStatusWeb(), access mode by default is Read FIX: RSS __init__.py will not crash anymore if no CS info provided BUGFIX: CS.getSiteTier now behaves correctly when a site is passed as a string *dirac-setup-site BUGFIX: fixed typos in the Script class name *Transformation FIX: Missing logger in the TaskManager Client (was using agent's one) NEW: Added UnitTest class for TaskManager Client *DIRAC API BUGFIX: Dirac.py. If /LocalSite/FileCatalog is not define the default Catalog was not properly set. FIX: Dirac.py - fixed __printOutput to properly interpret the first argument: 0:stdout, 1:stderr NEW: Dirac.py - added getConfigurationValue() method *Framework NEW: UsersAndGroups agent to synchronize users from VOMRS server. *dirac-install FIX: make Platform.py able to run with python2.3 to be used inside dirac-install FIX: protection against the old or pro links pointing to non-existent directories NEW: make use of the HTTP proxies if available FIX: fixed the logic of creating links to /opt/dirac directories to take into account webRoot subdirs *WorkloadManagement FIX: SiteDirector - change getVO() function call to getVOForGroup() *Core: FIX: Pfn.py - check the sanity of the pfn and catch the erroneous case *RequestManagement: BUGFIX: RequestContainer.isSubrequestDone() - return 0 if Done check fails *DataManagement NEW: FileCatalog - possibility to configure multiple FileCatalog services of the same type [v6r0p4] *Framework NEW: Pass the monitor down to the request RequestHandler FIX: Define the service location for the monitor FIX: Close some connections that DISET was leaving open [v6r0p3] *Framework FIX: ProxyManager - Registry.groupHasProperties() wasn't returning a result CHANGE: Groups without AutoUploadProxy won't receive expiration notifications FIX: typo dirac-proxy-info -> dirac-proxy-init in the expiration mail contents CHANGE: DISET - directly close the connection after a failed handshake [v6r0p2] *Framework FIX: in services logs change ALWAYS log level for query messages to NOTICE [v6r0p1] *Core BUGFIX: List.uniqueElements() preserves the other of the remaining elements *Framework CHANGE: By default set authorization rules to authenticated instead of all FIX: Use all required arguments in read access data for UserProfileDB FIX: NotificationClient - dropped LHCb-Production setup by default in the __getRPSClient() [v6r0] *Framework NEW: DISET Framework modified client/server protocol, messaging mechanism to be used for optimizers NEW: move functions in DIRAC.Core.Security.Misc to DIRAC.Core.Security.ProxyInfo CHANGE: By default log level for agents and services is INFO CHANGE: Disable the log headers by default before initializing NEW: dirac-proxy-init modification according to issue #29: -U flag will upload a long lived proxy to the ProxyManager If /Registry/DefaultGroup is defined, try to generate a proxy that has that group Replaced params.debugMessage by gLogger.verbose. Closes #65 If AutoUploadProxy = true in the CS, the proxy will automatically be uploaded CHANGE: Proxy upload by default is one month with dirac-proxy-upload NEW: Added upload of pilot proxies automatically NEW: Print info after creating a proxy NEW: Added setting VOMS extensions automatically NEW: dirac-proxy-info can also print the information of the uploaded proxies NEW: dirac-proxy-init will check that the lifetime of the certificate is less than one month and advise to renew it NEW: dirac-proxy-init will check that the certificate has at least one month of validity FIX: Never use the host certificate if there is one for dirac-proxy-init NEW: Proxy manager will send notifications when the uploaded proxies are about to expire (configurable via CS) NEW: Now the proxyDB also has a knowledge of user names. Queries can use the user name as a query key FIX: ProxyManager - calculate properly the dates for credentials about to expire CHANGE: ProxyManager will autoexpire old proxies, also auto purge logs CHANGE: Rename dirac-proxy-upload to dirac-admin-proxy-upload NEW: dirac-proxy-init will complain if the user certificate has less than 30 days CHANGE: SecurityLogging - security log level to verbose NEW: OracleDB - added Array type NEW: MySQL - allow definition of the port number in the configuration FIX: Utilities/Security - hash VOMS Attributes as string FIX: Utilities/Security - Generate a chain hash to discover if two chains are equal NEW: Use chain has to discover if it has already been dumped FIX: SystemAdministrator - Do not set a default lcg version NEW: SystemAdministrator - added Project support for the sysadmin CHANGE: SysAdmin CLI - will try to connect to the service when setting the host NEW: SysAdmin CLI - colorization of errors in the cli NEW: Logger - added showing the thread id in the logger if enabled *Configuration NEW: added getVOfromProxyGroup() utility NEW: added getVoForGroup() utility, use it in the code as appropriate NEW: added Registry and Operations Configuration helpers NEW: dirac-configuration-shell - a configuration script for CS that behaves like an UNIX shellCHANGE: CSAPI - added more functionality required by updated configuration console NEW: Added possibility to define LocalSE to any Site using the SiteLocalSEMapping section on the Operations Section NEW: introduce Registry/VO section, associate groups to VOs, define SubmitPools per VO FIX: CE2CSAgent - update the CEType only if there is a relevant info in the BDII *ReleaseManagement NEW: release preparations and installation tools based on installation packages NEW: dirac-compile-externals will try go get a DIRAC-free environment before compiling NEW: dirac-disctribution - upload command can be defined via defaults file NEW: dirac-disctribution - try to find if the version name is a branch or a tag in git and act accordingly NEW: dirac-disctribution - added keyword substitution when creating a a distribution from git FIX: Install tools won't write HostDN to the configuration if the Admin username is not set FIX: Properly set /DIRAC/Configuration/Servers when installing a CS Master FIX: install_site.sh - missing option in wget for https download: --no-check-certificate FIX: dirac-install-agent(service) - If the component being installed already has corresponding CS section, it is not overwritten unless explicitly asked for NEW: dirac-install functionality enhancement: start using the switches as defined in issue #26; CHANGE: dirac-install - write the defaults if any under defaults-.cfg so dirac-configure can pick it up FIX: dirac-install - define DYLD_LIBRARY_PATH ( for Mac installations ) NEW: dirac-install - put all the goodness under a function so scripts like lhcb-proxy-init can use it easily FIX: dirac-install - Properly search for the LcgVer NEW: dirac-install will write down the releases files in -d mode CHANGE: use new dirac_install from gothub/integration branch in install_site.sh NEW: Extensions can request custom external dependencies to be installed via pip when installing DIRAC. NEW: LCG bundle version can be defined on a per release basis in the releases.cfg NEW: dirac-deploy-scripts - when setting the lib path in the deploy scripts. Also search for subpaths of the libdir and include them NEW: Install tools - plainly separate projects from installations *Accounting CHANGE: For the WMSHistory type, send as JobSplitType the JobType CHANGE: Reduced the size of the max key length to workaround mysql max bytes for index problem FIX: Modified buckets width of 1week to 1 week + 1 day to fix summer time end week (1 hour more ) *WorkloadManagement CHANGE: SiteDirector - simplified executable generation NEW: SiteDirector - few more checks of error conditions NEW: SiteDirector - limit the queue max length to the value of MaxQueueLengthOption ( 3 days be default ) BUGFIX: SiteDirector - do not download pilot output if the flag getPilotOutput is not set NEW: JobDB will extract the VO when applying DIRAC/VOPolicy from the proper VO FIX: SSHTorque - retrieve job status by chunks of 100 jobs to avoid too long NEW: glexecComputingElement - allow glexecComputingElement to "Reschedule" jobs if the Test of the glexec fails, instead of defaulting to InProcess. Controlled by RescheduleOnError Option of the glexecComputingElement NEW: SandboxStore - create a different SBPath with the group included FIX: JobDB - properly treat Site parameter in the job JDL while rescheduling jobs NEW: JobSchedulingAgent - set the job Site attribute to the name of a group of sites corresponding to a SE chosen by the data staging procedure CHANGE: TimeLeft - call batch system commands with the ( default ) timeout 120 sec CHANGE: PBSTimeLeft - uses default CPU/WallClock if not present in the output FIX: PBSTimeLeft - proper handling of (p)cput parameter in the batch system output, recovery of the incomplete batch system output NEW: automatically add SubmitPools JDL option of the job owner's VO defines it NEW: JobManager - add MaxParametericJobs option to the service configuration NEW: PilotDirector - each SubmitPool or Middleware can define TargetGrids NEW: JobAgent - new StopOnApplicationFailure option to make the agent exiting the loop on application failure NEW: PilotAgentsDB - on demand retrieval of the CREAM pilot output NEW: Pilot - proper job ID evaluation for the OSG sites FIX: ComputingElement - fixed proxy renewal logic for generic and private pilots NEW: JDL - added %j placeholder in the JDL to be replaced by the JobID BUGFIX: DownloadInputData - bug fixed in the naming of downloaded files FIX: Matcher - set the group and DN when a request gets to the matcher if the request is not coming from a pilot FIX: Matcher = take into account JobSharing when checking the owner for the request CHANGE: PilotDirector, dirac-pilot - interpret -V flag of the pilot as Installation name *DataManagement FIX: FileCatalog/DiractoryLevelTree - consistent application of the max directory level using global MAX_LEVELS variable FIX: FileCatalog - Directory metadata is deleted together with the directory deletion, issue #40 CHANGE: FileCatalog - the logic of the files query by metadata revisited to increase efficiency FIX: LcgFileCatalog - use lfcthr and call lfcthr.init() to allow multithread try the import only once and just when LcgFileCatalogClient class is intantiated NEW: LcgFileCatalogClient - new version of getPathPermissions relying on the lfc_access method to solve the problem of multiple user DNs in LFC. FIX: StorageElement - get service CS options with getCSOption() method ( closes #97 ) FIX: retrieve FileCatalogs as ordered list, to have a proper default. CHANGE: FileCatalog - allow up to 15 levels of directories BUGFIX: FileCatalog - bug fixes in the directory removal methods (closes #98) BUGFIX: RemovalAgent - TypeError when getting JobID in RemovalAgent BUGFIX: RemovalAgent - put a limit to be sure the execute method will end after a certain number of iterations FIX: DownloadInputData - when files have been uploaded with lcg_util, the PFN filename might not match the LFN file name FIX: putting FTSMonitor web page back NEW: The default file catalog is now determined using /LocalSite/FileCatalog. The old behavior is provided as a fallback solution NEW: ReplicaManager - can now deal with multiple catalogs. Makes sure the surl used for removal is the same as the one used for registration. NEW: PoolXMLCatalog - added getTypeByPfn() function to get the type of the given PFN NEW: dirac-dms-ban(allow)-se - added possibility to use CheckAccess property of the SE *StorageManagement FIX: Stager - updateJobFromStager(): only return S_ERROR if the Status sent is not recognized or if a state update fails. If the jobs has been removed or has moved forward to another status, the Stager will get an S_OK and should forget about the job. NEW: new option in the StorageElement configuration "CheckAccess" FIX: Requests older than 1 day, which haven't been staged are retried. Tasks older than "daysOld" number of days are set to Failed. These tasks have already been retried "daysOld" times for staging. FIX: CacheReplicas and StageRequests records are kept until the pin has expired. This way the StageRequest agent will have proper accounting of the amount of staged data in cache. NEW: FTSCleaningAgent will allow to fix transient errors in RequestDB. At the moment it's only fixing Requests for which SourceTURL is equal to TargetSURL. NEW: Stager - added new command dirac-stager-stage-files FIX: Update Stager code in v6 to the same point as v5r13p37 FIX: StorageManager - avoid race condition by ensuring that Links=0 in the query while removing replicas *RequestManagement FIX: RequestDBFile - get request in chronological order (closes issue #84) BUGFIX: RequestDBFile - make getRequest return value for getRequest the same as for *ResourceStatusSystem NEW: Major code refacoring. First refactoring of RSS's PEP. Actions are now function defined in modules residing in directory "Actions". NEW: methods to store cached environment on a DB and ge them. CHANGE: command caller looks on the extension for commands. CHANGE: RSS use now the CS instead of getting info from Python modules. BUGFIX: Cleaned RSS scripts, they are still prototypes CHANGE: PEP actions now reside in separate modules outside PEP module. NEW: RSS CS module add facilities to extract info from CS. CHANGE: Updating various RSS tests to make them compatible with changes in the system. NEW: CS is used instead of ad-hoc configuration module in most places. NEW: Adding various helper functions in RSS Utils module. These are functions used by RSS developers, including mainly myself, and are totally independant from the rest of DIRAC. CHANGE: Mostly trivial changes, typos, etc in various files in RSS CHANGE: TokenAgent sends e-mails with current status *Transformation CHANGE: allow Target SE specification for jobs, Site parameter is not set in this case CHANGE: TransformationAgent - add new file statuses in production monitoring display CHANGE: TransformationAgent - limit the number of files to be treated in TransformationAgent for replication and removal (default 5000) BUGFIX: TransformationDB - not removing task when site is not set BUGFIX: TransformationCleaningAgent - archiving instead of cleaning Removal and Replication transformations FIX: TransformationCleaningAgent - kill jobs before deleting them *Workflow NEW: allow modules to define Input and Output parameters that can be used instead of the step_commons/workflow_commons (Workflow.py, Step.py, Module.py) *Various fixes BUGFIX: Mail.py uses SMTP class rather than inheriting it FIX: Platform utility will properly discover libc version even for the new Ubuntu FIX: Removed old sandbox and other obsoleted components