How Do I View the Debugging Information Printed Using System.out.println or Export the Debugging Information to a Specified File?

Question

System.out.println is added to the Flink service codes for printing debugging information. How do I view the debugging log? How do I export service logs to a specified file to differentiate service logs from run logs?

Answer

All run logs of Flink are printed to the local directory of Yarn. By default, all logs are exported to taskmanager.log in the local directory of Yarn container. All logs generated by invoking System.out will be exported to the taskmanager.out file. You can perform as follows to view the logs:

Log in to the native Flink web page.
Choose Task Managers > Logs or Task Managers > Stdout on the left to view log information.

Configure the function of printing service logs and Task Manager run logs separately:

If service logs and Task Manager run logs are separately printed, service logs are not exported to the taskmanager.log file and cannot be viewed on the web page.

Modify the configuration file logback.xml in the conf directory of the client. Add the following log configuration information to the file. Modify the information in bold according to the actual situation.

<appender name="TEST" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/path/test.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
            <fileNamePattern>/path/test.log.%i</fileNamePattern>
            <minIndex>1</minIndex>
            <maxIndex>20</maxIndex>
        </rollingPolicy>
        <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
            <maxFileSize>20MB</maxFileSize>
        </triggeringPolicy>
        <encoder>
            <pattern>%d{"yyyy-MM-dd HH:mm:ss,SSS"} | %m %n</pattern>
        </encoder>
    </appender>

    <logger name="com.huawei.bigdata.flink.examples" additivity="false">
        <level value="INFO"/>
        <appender-ref ref="TEST"/>
    </logger>

Run yarn-session.sh to submit the task.

If the configuration file logback.xml contains <file>/path/test.log</file>, ensure that the user (configured flink-conf.yaml) used for running the task has the write and read permissions on the directory.