Quartz recovery 及misfired机制的源码分析

朴灿烈づ我的快乐病毒、 2022-06-16 13:14 1121阅读 0赞

首先要明确的是：quartz如果在执行具体任务时，在任务执行过程中抛出异常，那么不作任何处理，这是使用者程序本身的问题，不需要框架处理。

下面介绍quartz中的两大类异常情况：

misfired 哑火（\*注：笔者自己直译）

fail-over 故障转移

### 1.misfired 哑火 ###

哑火顾名思义，就是quartz在应该触发（fire）trigger的时候未能及时将其触发，这将导致trigger的下次触发时间落在当前时间之前，那么按照正常的quartz调度流程，该trigger就再也没有机会被调度了。由于一个调度器实例在每次调度的过程中都会有一定的睡眠时间，存在在一段时间内所有调度器实例都在睡眠，而无人触发调度的潜在可能。于是调度器需要每隔一段时间（15s~60s）查看一次各trigger的nextfiretime，检查出否有tirgger的下一次触发落在了当前时间之前足够长的时间，在这里系统设定了一个60s的域（misfireThreshold），当一个trigger的下一次触发时间早于当前时间60s之外时，调度器判定该触发器misfired，在发现有触发器哑火之后启动相应的流程回复trigger至正常状态。上述这些过程是在调度器初始化时与主调度线程类quartzSchedulerThread同时开启的一个线程类MisfireHandler中进行的。

下面是quartz检测misfired的逻辑：

下面是quartz对misfire处理的关键代码：

对trigger哑火处理的最关键一点在于针对不同策略对trigger的nextfiretime进行设定，这一过程对于不同的trigger类型有不同的策略供选择。

下面是各种不同triigger对应的不同misfire策略（摘自网络）：   
CronTrigger

withMisfireHandlingInstructionDoNothing   
——不触发立即执行   
——等待下次Cron触发频率到达时刻开始按照Cron频率依次执行

withMisfireHandlingInstructionIgnoreMisfires   
——以错过的第一个频率时间立刻开始执行   
——重做错过的所有频率周期后   
——当下一次触发频率发生时间大于当前时间后，再按照正常的Cron频率依次执行

withMisfireHandlingInstructionFireAndProceed(默认)   
——以当前时间为触发频率立刻触发一次执行   
——然后按照Cron频率依次执行

withMisfireHandlingInstructionFireNow   
——以当前时间为触发频率立即触发执行   
——执行至FinalTIme的剩余周期次数   
——以调度或恢复调度的时刻为基准的周期频率，FinalTime根据剩余次数和当前时间计算得到   
——调整后的FinalTime会略大于根据starttime计算的到的FinalTime值

withMisfireHandlingInstructionIgnoreMisfires   
——以错过的第一个频率时间立刻开始执行   
——重做错过的所有频率周期   
——当下一次触发频率发生时间大于当前时间以后，按照Interval的依次执行剩下的频率   
——共执行RepeatCount+1次

withMisfireHandlingInstructionNextWithExistingCount   
——不触发立即执行   
——等待下次触发频率周期时刻，执行至FinalTime的剩余周期次数   
——以startTime为基准计算周期频率，并得到FinalTime   
——即使中间出现pause，resume以后保持FinalTime时间不变

withMisfireHandlingInstructionNowWithExistingCount（默认）   
——以当前时间为触发频率立即触发执行   
——执行至FinalTIme的剩余周期次数   
——以调度或恢复调度的时刻为基准的周期频率，FinalTime根据剩余次数和当前时间计算得到   
——调整后的FinalTime会略大于根据starttime计算的到的FinalTime值

withMisfireHandlingInstructionNextWithRemainingCount   
——不触发立即执行   
——等待下次触发频率周期时刻，执行至FinalTime的剩余周期次数   
——以startTime为基准计算周期频率，并得到FinalTime   
——即使中间出现pause，resume以后保持FinalTime时间不变

withMisfireHandlingInstructionNowWithRemainingCount   
——以当前时间为触发频率立即触发执行   
——执行至FinalTIme的剩余周期次数   
——以调度或恢复调度的时刻为基准的周期频率，FinalTime根据剩余次数和当前时间计算得到

——调整后的FinalTime会略大于根据starttime计算的到的FinalTime值

MISFIRE\_INSTRUCTION\_RESCHEDULE\_NOW\_WITH\_REMAINING\_REPEAT\_COUNT   
——此指令导致trigger忘记原始设置的starttime和repeat-count   
——触发器的repeat-count将被设置为剩余的次数   
——这样会导致后面无法获得原始设定的starttime和repeat-count值

### 2.fail-over 故障转移 ###

quartz考虑的另一个问题是运行时的系统崩溃，在集群中如果有一个节点突然崩溃，那么它所执行的任务会被首先发现其崩溃的节点接手，重新执行，换句话说就是把故障节点的工作转移到其他节点上，简称故障转移。recovery机制工作在集群环境中，执行recovery工作的线程类叫做ClusterManager，该线程类同样是在调度器初始化时就开启运行了。这个线程类在运行期间每15s进行一次check in操作，所谓check in，就是在数据库的QRTZ2\_SCHEDULER\_STATE表中更新该调度器对应的LAST\_CHECKIN\_TIME字段为当前时间，并且查看其他调度器实例的该字段有没有发生停止更新的情况，如果检查到有调度器的check in time比当前时间要早约15s（视具体的执行预配置情况而定），那么就判定该调度实例需要recover，随后会启动该调度器的recovery机制，获取目标调度器实例正在触发的trigger，并针对每一个trigger临时添加一各对应的仅执行一次的simpletrigger。等到调度流程扫描trigger时，这些trigger会被触发，这样就成功的把这些未完整执行的调度以一种特殊trigger的形式纳入了普通的调度流程中，只要调度流程在正常运行，这些被recover的trigger就会很快被触发并执行。

下面的代码是ClusterManager线程类的run方法，可以看到，该线程类不断地在调用manage方法，该方法中包含了check in与recover的逻辑。

manage方法主要调用doCheckIn方法，该方法中承载的check in与recover的详细逻辑：

在代码中对第一次check in的操作比较令人困惑，不管是不是第一次check in似乎都需要调用clusterCheckIn方法，而该方法内部又调用了findFailedInstances方法，见代码：

而findFailedInstances方法中有一个处理无主trigger的逻辑，无主trigger是说在QRTZ2\_FIRED\_TRIGGERS表中如果一条记录的调度器id在QRTZ2\_SCHEDULER\_STATE表中找不到相应的记录，那么这条trigger触发就成为一个无主的记录。这种记录只能查询到其调用者id，而无法知道QRTZ2\_SCHEDULER\_STATE表中可以查到的其他字段，需要系统做特殊处理（TODO:如何处理）。

上述的情况需要在五新的QRTZ2\_SCHEDULER\_STATE记录插入时进行，所以在doCheckin中的安排仅仅是为了让调度器在自身数据要加入进QRTZ2\_SCHEDULER\_STATE表之前检查一遍是否有这种无主数据，并做处理。

回到doCheckin方法中，得到需要recover的调度器实例列表，启动recover流程。clusterRecover的代码如下：

<table style=""> 
 <tbody> 
  <tr style="font-size:14px; line-height:1.3em"> 
   <td style="font-size:14px; line-height:1.3em"> 
    <div title="Hint: double-click to select code" style="font-size:16px; line-height:1.7em"> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">protected</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">void</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">clusterRecover(Connection conn, List&lt;SchedulerStateRecord&gt; failedInstances)</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">throws</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">JobPersistenceException { </code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">if</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">(failedInstances.size() &gt;&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">) { </code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">long</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">recoverIds = System.currentTimeMillis();</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">logWarnIfNonZero(failedInstances.size(),</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">"ClusterManager: detected "</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">+ failedInstances.size()</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">+&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">" failed or restarted instances."</code>&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">);</code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">try</code> &nbsp;&nbsp; &nbsp; 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">{ </code> 
     </div> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">//迭代需要recover的SchedulerStateRecord列表</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">for</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">(SchedulerStateRecord rec : failedInstances) { </code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">getLog().info(</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">"ClusterManager: Scanning for instance \""</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">+ rec.getSchedulerInstanceId()</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">+&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">"\"'s failed in-progress jobs."</code>&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">);</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">//读取该实例遗留的触发记录</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">List&lt;FiredTriggerRecord&gt; firedTriggerRecs = getDelegate()</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">.selectInstancesFiredTriggerRecords(conn,</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">rec.getSchedulerInstanceId());</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">int</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">acquiredCount =&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">;</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">int</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">recoveredCount =&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">;</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">int</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">otherCount =&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">;</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">Set&lt;TriggerKey&gt; triggerKeys =&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">new</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">HashSet&lt;TriggerKey&gt;();</code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">//迭代触发记录</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">for</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">(FiredTriggerRecord ftRec : firedTriggerRecs) { </code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">TriggerKey tKey = ftRec.getTriggerKey();</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">JobKey jKey = ftRec.getJobKey();</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">triggerKeys.add(tKey);</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">// release blocked triggers..</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">if</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">(ftRec.getFireInstanceState().equals(STATE_BLOCKED)) { </code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">getDelegate()</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">.updateTriggerStatesForJobFromOtherState(</code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">conn, jKey,</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">STATE_WAITING, STATE_BLOCKED);</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">}&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">else</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">if</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">(ftRec.getFireInstanceState().equals(STATE_PAUSED_BLOCKED)) { </code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">getDelegate()</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">.updateTriggerStatesForJobFromOtherState(</code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">conn, jKey,</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">STATE_PAUSED, STATE_PAUSED_BLOCKED);</code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">}</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">// release acquired triggers..</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">if</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">(ftRec.getFireInstanceState().equals(STATE_ACQUIRED)) { </code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">getDelegate().updateTriggerStateFromOtherState(</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">conn, tKey, STATE_WAITING,</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">STATE_ACQUIRED);</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">acquiredCount++;</code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">//如果trigger指定的job策略为需要recovery，那么执行recovery</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">}&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">else</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">if</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">(ftRec.isJobRequestsRecovery()) { </code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">// handle jobs marked for recovery that were not fully</code> 
     </div> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">// executed..</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">if</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">(jobExists(conn, jKey)) { </code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">@SuppressWarnings</code>&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">(</code>&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">"deprecation"</code>&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">)</code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">//新建一次性的simpletrigger，用以重新执行需要recover的tirgger</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">SimpleTriggerImpl rcvryTrig =&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">new</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">SimpleTriggerImpl(</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">"recover_"</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">+ rec.getSchedulerInstanceId()</code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">+&nbsp;</code> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">"_"</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">+ String.valueOf(recoverIds++),</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">Scheduler.DEFAULT_RECOVERY_GROUP,</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">new</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">Date(ftRec.getScheduleTimestamp()));</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">rcvryTrig.setJobName(jKey.getName());</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">rcvryTrig.setJobGroup(jKey.getGroup());</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">rcvryTrig.setMisfireInstruction(SimpleTrigger.MISFIRE_INSTRUCTION_IGNORE_MISFIRE_POLICY);</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">rcvryTrig.setPriority(ftRec.getPriority());</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">//读取jobDataMap</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">JobDataMap jd = getDelegate().selectTriggerJobDataMap(conn, tKey.getName(), tKey.getGroup());</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">jd.put(Scheduler.FAILED_JOB_ORIGINAL_TRIGGER_NAME, tKey.getName());</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">jd.put(Scheduler.FAILED_JOB_ORIGINAL_TRIGGER_GROUP, tKey.getGroup());</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">jd.put(Scheduler.FAILED_JOB_ORIGINAL_TRIGGER_FIRETIME_IN_MILLISECONDS, String.valueOf(ftRec.getFireTimestamp()));</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">jd.put(Scheduler.FAILED_JOB_ORIGINAL_TRIGGER_SCHEDULED_FIRETIME_IN_MILLISECONDS, String.valueOf(ftRec.getScheduleTimestamp()));</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">rcvryTrig.setJobDataMap(jd);</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">rcvryTrig.computeFirstFireTime(</code>&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">null</code>&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">);</code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">//持久化一次性的simpleTrigger，使得这个trigger可以被正常的调度流程触发。</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">storeTrigger(conn, rcvryTrig,&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">null</code>&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">,&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">false</code>&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">,</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">STATE_WAITING,&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">false</code>&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">,&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">true</code>&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">);</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">recoveredCount++;</code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">}&nbsp;</code> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">else</code> &nbsp;&nbsp; &nbsp; 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">{ </code> 
     </div> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">getLog()</code> 
     </div> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">.warn(</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">"ClusterManager: failed job '"</code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">+ jKey</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">+&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">"' no longer exists, cannot schedule recovery."</code>&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">);</code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">otherCount++;</code> 
     </div> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">}</code> 
     </div> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">}&nbsp;</code> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">else</code> &nbsp;&nbsp; &nbsp; 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">{ </code> 
     </div> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">otherCount++;</code> 
     </div> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">}</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">// free up stateful job's triggers</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">if</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">(ftRec.isJobDisallowsConcurrentExecution()) { </code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">getDelegate()</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">.updateTriggerStatesForJobFromOtherState(</code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">conn, jKey,</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">STATE_WAITING, STATE_BLOCKED);</code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">getDelegate()</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">.updateTriggerStatesForJobFromOtherState(</code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">conn, jKey,</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">STATE_PAUSED, STATE_PAUSED_BLOCKED);</code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">}</code> 
     </div> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">}</code> 
     </div> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">//删除fired_trigger表中的trigger触发记录</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">getDelegate().deleteFiredTriggers(conn,</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">rec.getSchedulerInstanceId());</code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">// Check if any of the fired triggers we just deleted were the last fired trigger</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">// records of a COMPLETE trigger.</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">int</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">completeCount =&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">;</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">for</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">(TriggerKey triggerKey : triggerKeys) { </code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">if</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">(getDelegate().selectTriggerState(conn, triggerKey).</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">equals(STATE_COMPLETE)) { </code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">List&lt;FiredTriggerRecord&gt; firedTriggers =</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">getDelegate().selectFiredTriggerRecords(conn, triggerKey.getName(), triggerKey.getGroup());</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">if</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">(firedTriggers.isEmpty()) { </code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">if</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">(removeTrigger(conn, triggerKey)) { </code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">completeCount++;</code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">}</code> 
     </div> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">}</code> 
     </div> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">}</code> 
     </div> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">}</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">logWarnIfNonZero(acquiredCount,</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">"ClusterManager: ......Freed "</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">+ acquiredCount</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">+&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">" acquired trigger(s)."</code>&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">);</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">logWarnIfNonZero(completeCount,</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">"ClusterManager: ......Deleted "</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">+ completeCount</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">+&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">" complete triggers(s)."</code>&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">);</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">logWarnIfNonZero(recoveredCount,</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">"ClusterManager: ......Scheduled "</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">+ recoveredCount</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">+&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">" recoverable job(s) for recovery."</code>&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">);</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">logWarnIfNonZero(otherCount,</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">"ClusterManager: ......Cleaned-up "</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">+ otherCount</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">+&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">" other failed job(s)."</code>&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">);</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">if</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">(!rec.getSchedulerInstanceId().equals(getInstanceId())) { </code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">getDelegate().deleteSchedulerState(conn,</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">rec.getSchedulerInstanceId());</code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">}</code> 
     </div> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">}</code> 
     </div> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">}&nbsp;</code><code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">catch</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">(Throwable e) { </code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">throw</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">new</code>&nbsp;&nbsp;&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">JobPersistenceException(</code>&nbsp;<code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">"Failure recovering jobs: "</code></p> 
     <p style="margin:0px 0px 0.75em; font-size:16px; line-height:1.7em; text-indent:1em"> <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">+ e.getMessage(), e);</code></p> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">}</code> 
     </div> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">}</code> 
     </div> 
     <div style="font-size:16px; line-height:1.7em"> 
      <code style="padding:2px 4px; font-family:Menlo,Monaco,Consolas,'Courier New',monospace; color:rgb(51,51,51)!important; background-color:rgb(247,247,249); border:none; white-space:normal; font-weight:600; font-size:14px">}</code> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

可以看到，最终一个需要recover的节点，其未执行完整的任务最终会被其他节点已新建临时trigger的形式重新执行。这一系列的机制正是保证quartz稳定性的可靠保证。