-i"PSrSSKr"SS5r"SS\5r"SS\5rg) z'Stochastic optimization methods for MLPNc4\rSrSrSrS SjrSrSrSrSr g) BaseOptimizer aBase (Stochastic) gradient descent optimizer Parameters ---------- learning_rate_init : float, default=0.1 The initial learning rate used. It controls the step-size in updating the weights Attributes ---------- learning_rate : float the current learning rate c0Xl[U5UlgN)learning_rate_initfloat learning_rate)selfrs `/var/www/html/venv/lib/python3.13/site-packages/sklearn/neural_network/_stochastic_optimizers.py__init__BaseOptimizer.__init__s"4"#56cfURU5n[SU5U5H upEXE- nM g)aUpdate parameters with given gradients Parameters ---------- params : list of length = len(coefs_) + len(intercepts_) The concatenated list containing coefs_ and intercepts_ in MLP model. Used for initializing velocities and updating params grads : list of length = len(params) Containing gradients with respect to coefs_ and intercepts_ in MLP model. So length should be aligned with params c3$# UHov M g7fr).0ps r .BaseOptimizer.update_params..*s!4V!VsN) _get_updateszip)r paramsgradsupdatesparamupdates r update_paramsBaseOptimizer.update_paramss4##E* !4V!4g>ME OE?rcg)zXPerform update to learning rate and potentially other states at the end of an iteration Nrr time_steps r iteration_endsBaseOptimizer.iteration_ends-s rc.U(a[US-5 g)zDecides whether it is time to stop training Parameters ---------- msg : str Message passed in for verbose output verbose : bool Print message to stdin if True Returns ------- is_stopping : bool True if training needs to stop Stopping.T)printr msgverboses r trigger_stoppingBaseOptimizer.trigger_stopping3s  # $ %r)r rN)皙?) __name__ __module__ __qualname____firstlineno____doc__r rr#r+__static_attributes__rrr rr s 7" rrcL^\rSrSrSrSU4SjjrSrSrSrSr U=r $) SGDOptimizerHaCStochastic gradient descent optimizer with momentum Parameters ---------- params : list, length = len(coefs_) + len(intercepts_) The concatenated list containing coefs_ and intercepts_ in MLP model. Used for initializing velocities and updating params learning_rate_init : float, default=0.1 The initial learning rate used. It controls the step-size in updating the weights lr_schedule : {'constant', 'adaptive', 'invscaling'}, default='constant' Learning rate schedule for weight updates. -'constant', is a constant learning rate given by 'learning_rate_init'. -'invscaling' gradually decreases the learning rate 'learning_rate_' at each time step 't' using an inverse scaling exponent of 'power_t'. learning_rate_ = learning_rate_init / pow(t, power_t) -'adaptive', keeps the learning rate constant to 'learning_rate_init' as long as the training keeps decreasing. Each time 2 consecutive epochs fail to decrease the training loss by tol, or fail to increase validation score by tol if 'early_stopping' is on, the current learning rate is divided by 5. momentum : float, default=0.9 Value of momentum used, must be larger than or equal to 0 nesterov : bool, default=True Whether to use nesterov's momentum or not. Use nesterov's if True power_t : float, default=0.5 Power of time step 't' in inverse scaling. See `lr_schedule` for more details. Attributes ---------- learning_rate : float the current learning rate velocities : list, length = len(params) velocities that are used to update params c>[TU]U5 X0lX@lXPlX`lUVs/sHn[ R"U5PM snUlgs snfr) superr lr_schedulemomentumnesterovpower_tnp zeros_like velocities) r rrr9r:r;r<r __class__s r r SGDOptimizer.__init__xsJ +,&   =CDVE2==/VDDs AcURS:Xa.[UR5US-UR-- Ulgg)zPerform updates to learning rate and potential other states at the end of an iteration Parameters ---------- time_step : int number of training samples trained on so far, used to update learning rate for 'invscaling' invscalingN)r9r rr<r r!s r r#SGDOptimizer.iteration_endss?   | +d--.)a-DLL1PP   ,rc URS:waU(a[US-5 gURS::aU(a[US-5 gU=RS-slU(a[USUR--5 g) Nadaptiver&Tgư>z# Learning rate too small. Stopping.g@z Setting learning rate to %fF)r9r'r r(s r r+SGDOptimizer.trigger_stoppingsu   z )cL()    %cAAB c!  #69K9KKK Lrch[URU5VVs/sH$up#URU-URU-- PM& nnnX@lUR(aG[URU5VVs/sH$up#URU-URU-- PM& nnnU$s snnfs snnf)VGet the values used to update params with given gradients Parameters ---------- grads : list, length = len(coefs_) + len(intercepts_) Containing gradients with respect to coefs_ and intercepts_ in MLP model. So length should be aligned with params Returns ------- updates : list, length = len(grads) The values to add to params )rr?r:r r;)r rvelocitygradrs r rSGDOptimizer._get_updatess #&doou"= "= MMH $t'9'9D'@ @"=  " =='*$//5&A&ANH (4+=+=+DD&A   s +B(8+B.)r r9r:r;r<r?)r-constant?Tg?) r.r/r0r1r2r r#r+rr3 __classcell__r@s@r r5r5Hs4-dE"  rr5c8^\rSrSrSrSU4SjjrSrSrU=r$) AdamOptimizeraJStochastic gradient descent optimizer with Adam Note: All default values are from the original Adam paper Parameters ---------- params : list, length = len(coefs_) + len(intercepts_) The concatenated list containing coefs_ and intercepts_ in MLP model. Used for initializing velocities and updating params learning_rate_init : float, default=0.001 The initial learning rate used. It controls the step-size in updating the weights beta_1 : float, default=0.9 Exponential decay rate for estimates of first moment vector, should be in [0, 1) beta_2 : float, default=0.999 Exponential decay rate for estimates of second moment vector, should be in [0, 1) epsilon : float, default=1e-8 Value for numerical stability Attributes ---------- learning_rate : float The current learning rate t : int Timestep ms : list, length = len(params) First moment vectors vs : list, length = len(params) Second moment vectors References ---------- :arxiv:`Kingma, Diederik, and Jimmy Ba (2014) "Adam: A method for stochastic optimization." <1412.6980> c>[TU]U5 X0lX@lXPlSUlUVs/sHn[ R"U5PM snUlUVs/sHn[ R"U5PM snUl gs snfs snf)Nr) r8r beta_1beta_2epsilontr=r>msvs)r rrrVrWrXrr@s r r AdamOptimizer.__init__so +,   5;) r.r/r0r1r2r rr3rPrQs@r rSrSs+\SW =  rrS)r2numpyr=rr5rSrrr rds5- <<~z=zzZMZr